14 ms·
The Evolution of Container Usage at Netflix
- hueving 10y ago>The theme that underlies all these improvements is developer innovation velocity I can't wait until this becomes buzzword de jour and startups start using it in their product descriptions. Then someone needs to start talking about products to "enable developer innovation acceleration" to outpace these crufty companies stuck at 25 kph.
- geodel 10y agoYup. I think further down someone will 'democratize innovation velocity and acceleration'
- chii 10y ago"unlock innovation potential"
- billfor 10y agoYou have to do it holistically though.
- RangerScience 10y agoHolistically-caused emergent innovation velocity and acceleration? ...The scary thing is I think that makes sense, actually. I can understand that sentence as a thing I would want - Twiddle with your company culture so that individuals come up with (and make) new ideas, in such a way that as time goes on, their ability to do that grows...? Makes sense to go at it from a systems view (holistic), rather than components view. How about - Homeopathic synergies for creative empowerment and evolution? That sounds properly almost, but not actually, sensical.
- RightMillennial 10y agoBut watch out for the innovative jerks. They may mess up the acceleration.
- nnutter 10y agoAs long as they include direction and magnitude it's OK with me.
- politician 10y agoThere's a whole list of higher-order developer innovation derivatives for marketing types to explore - jerk, snap, ...
- deleted 10y ago[deleted]
- nailer 10y agoThey're killing performance (one of the main reasons to use containers) and adding a massive extra layer of management if they're running containers on EC2. I suspect Netflix are too wedded to AWS (which is weird as Amazon is their biggest threat) but Triton or Red Shift (both of which actually isolate containers using SmartOS and SELinux respectively) make way more sense for other people who want to use the blazing fast IO speed of containers on bare metal.
- richardwhiuk 10y agoOr use something like GKE where the containers should be running directly on hardware, so you only have one layer.
- nailer 10y ago"Bare-metal" means running directly on hardware. But yes, GKE would also join Triton and Red Shift (provided it has similar container isolation capabilities).
- nailer 10y agoActually, as 0x74696d mentions below, GKE is also not bare metal. Doing a little research: GKE runs on https://cloud.google.com/container-optimized-os/docs/ https://cloud.google.com/container-optimized-os/docs/ which is designed for GCE, which runs on KVM: https://cloud.google.com/compute/docs/faq https://cloud.google.com/compute/docs/faq So it looks like just Triton and Red Shift. Disclaimer: I used to work at Red Hat, which means I like the people behind Red Shift, but that also means I hate the people behind Triton as part of the early 2000s Linux/Solaris wars.
- 0x74696d 10y agoGKE clusters run on VMs, not on bare metal.
- lima 10y agoContainers don't actually run directly on hardware with GCE - there's still a virtualization layer in-between. I'm 99% sure that Google runs one VM per container because that's the only way to make it safe. Anything else would be insane.
- bertlequant 10y agoI would've imagined Netflix had their own hardware, both compute and storage.
- neom 10y agoadrian cockcroft, who now works for aws, is known for a lot of his very early forward thinking approach to distributed applications and was responsible for their cloud architecture. I highly recommend checking out his medium and twitter, he's a brilliant man.
- nailer 10y agoI've heard they're looking at 'architecture in a box' solutions, which would allow them to abstract designs from specific cloud providers and perform cloud arbitrage; eg, if Cloud Provider X can run the app at 14c/hour, and Cloud Provider Y can run the app at 22c/hour, then they can just deploy the entire architecture on Cloud Provider Y.
- jjnoakes 10y agoWhy would they choose the more expensive one?
- user5994461 10y agoAnother simple problem that's solved: Auto scaling of stateless services across thousands of machines, in multiple datacenter and cloud providers all around the world. https://www.nomadproject.io/ https://www.nomadproject.io/ There are also kubernetes and mesosphere to do similar thing, but they are harder to use, you can't learn them over a week end.
- yeukhon 10y agoNetflix Engineering team amazes me a lot. They literally took all the available apis, and build their own platform despite some of the features are already in the AWS offerings. I suppose they did it mainly because the native service isn't flexible and robust enough for their use cases. BTW, their opening positions are always prefixed with "senior" title but I guess that makes sense; Netflix builds pretty much everything from scratch under time constraint.
- tybit 10y agoThey specifically only hire seniors, they don't want to provide positions for juniors as they would prefer to pay more for more seniors than pay indirectly for training juniors. I think this reflects poorly on them, just as much as companies that use OSS but don't contribute anything back.
- jhall1468 10y agoHow does that reflect poorly on them? They are willing to sacrifice potentially great hires because they'd rather make velocity their focal point. It's a trade-off.
- barrkel 10y agoI think it's a refreshing change from the ageism of much of the rest of the industry. It probably gives them a compounding competitive advantage.
- JustSomeNobody 10y agoI guess "experience discrimination" is a thing now? Seriously, if a company only wants to hire experienced folk, then so be it as long as they don't exercise REAL discrimination.
- curun1r 10y agoI personally find it short-sighted since I find mentorship opportunities to be a hugely rewarding part of my job. I wouldn't want to take a job in an environment with no junior developers since it would deprive me of that perk. I'd imagine Netflix misses out on other senior engineers for the same reason. It's also interesting considering a quote I remember from Google about their preference for junior engineers..."fewer bad habits to break."
- brianwawok 10y agoHow does this compare to k8s? Seems like it has everything plus a layer of scheduling and batch jobs on top?
- scurvy 10y agoDoesn't kubernetes run an overlay network (at least in some scenarios)?
- brianwawok 10y agoThat is true, it has more fancy network stuff, but less scheduling stuff. This space is getting busy in the last few years..
- gtirloni 10y agoAn overlay network is not a requirement. The only requirement is that pods (collection of containers) should be able to communicate with each other directly without NAT. Each pod gets an IP address in the container network. Overlay network technologies (flannel, weave, calico, etc) are popular but they aren't mandatory. You can implement it using hardware switches and VLANs if you wish.
- scurvy 10y agoSorry, but what are you inferring by hardware switches? Something not an overlay? Something that switches VXLAN in hardware?
- wahnfrieden 10y agoI wonder how this compares with the recently launched AWS Batch.
- dominotw 10y ago> We implemented multi-tenant isolation (CPU, memory, disk, networking and security) using a combination of Linux, Docker and our own isolation technology. Curious what their 'own isolation technology' does that docker doesn't. Also, what does Fenzo do that marathon doesn't . Looks like Fenzo sits on top of marathon and sends it some sort of recommendations for scheduling. I need to find a good example of what this is actually doing.
- skrater 10y agoJustifying high engineering salaries, i see this a lot in great companies.
- colanderman 10y agoNetflix is the last place I'd expect to do this. They're famous for letting people go when they're no longer needed.
- macNchz 10y agoThat sounds like a great incentive to build big proprietary systems that depend on your presence...
- draw_down 10y agoYeah, boo on those evil developers who want to make money and work on interesting stuff. What are you, their CFO?
- tyingq 10y ago>Curious what their 'own isolation technology' does that docker doesn't. Probably related to that "and security" part. That isn't currently docker's strong suit.
- lowbloodsugar 10y agoFenzo is a java library for deciding how to allocate tasks when offered resources. It makes implementing Mesos frameworks easier because it turns out the question: "Given an offer of X resources, and a list of tasks that need running, what is the best use of the offered resources?" is actually quite hard. (3D knapsack hard). "Apache Mesos frameworks match and assign resources to pending tasks. Fenzo presents a plugin-based, Java library that facilitates scheduling resources to tasks by using a variety of possible scheduling objectives, such as bin packing, balancing across resource abstractions (such as AWS availability zones or data center racks), resource affinity, and task locality." [1] Marathon is a Mesos framework for scheduling long running applications (like rest services) and keeping them running. If what you want to do is serve http traffic then Marathon does the job (although the stand-alone UI is now deprecated and will only bug-fixed for "the next few months", so you'd better like the full DC/OS offering). Titus appears to combine the functionality of Marathon, plus the ability to run batch jobs. I wondered if Titus was a fork of Marathon with new bits, but that doesn't appear to be the case. I believe it deals with one glaring flaw in Mesos, which is that frameworks all independently calculate the best use of the offered resources. When compute becomes available, Mesos makes offers to frameworks, with some basic logic such as making offers to frameworks currently consuming the least. But that means that there is no way to customize the resource allocation across different use cases (e.g. between REST APIs, one-off tasks and Spark clusters). It'd be great if Fenzo did "sit on top of Marathon", so I could customize how it schedules based on the "bigger picture". Titus avoids the problem because it schedules everything. [1] https://github.com/Netflix/Fenzo https://github.com/Netflix/Fenzo
- sandGorgon 10y ago> Today, we are in the process of rebuilding how we deploy device-specific server-side logic to our API tier leveraging single core optimized NodeJS servers Is this the core Netflix API ? have they moved from java - previously, their entire open source contributions were around java (https://netflix.github.io/ https://netflix.github.io/). Hystrix repo was updated barely a day ago. For me, this is more interesting than the VM part.
- meddlepal 10y agoThey use a mix of everything but most of the back end microservices are JVM tech - Java, Groovy and I heard they were using some Kotlin awhile ago.
- aaronblohowiak 10y agoIt is a splitting of the device-specific stuff from the general coordination stuff. So the "core API" as you put it remains in java while letting the device teams write their code in node (much of our UI teams are experts in JS already). Hystrix is important and won't be going away any time soon. More info: https://www.slideshare.net/mobile/KatharinaProbst/the-new-netflix-api https://www.slideshare.net/mobile/KatharinaProbst/the-new-ne... Disclaimer: I'm on paternity leave and not on those teams, but we've talked publicly about this stuff recently..
- sandGorgon 10y agoThis is very interesting. And very indicative of convergence. If someone at the same of Netflix has a strong drive towards cross pollination of talent from device to server, then I suppose the js ecosystem is far more successful than I thought.
- RcouF1uZ4gsC 10y agoNetflix seems so over engineered to me. They basically have a catalog of a few thousand movies that are negotiated months in advance of actual use. Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box. In regard to the analytics, they have 100 million subscribers. Let's say each subscriber watches an average of 100 episodes/movies a day. For each watch you record subscriber ID, movie ID, start time, stop time and get 32 bytes * 100 * 100,000,000 = 320 gigabytes of data per day total. I am pretty sure that you could get a commercial database and and business intelligence package that could support the type of analytics you need (mainly clustering analysis) at that scale. A national grocery chain, probably has a similar amount of data ingestion and a similar analytics need. In addition, I have subscribed to multiple Netflix type services and I have never weighed the quality of suggestions very high, giving much more weight to the functionality of the client, lack of ads, and large catalog of good movies. As evidence that this is a solved engineering problem, just look at the number of similar movie services: Amazon, Google play, Hulu, PlayStation Vue, Pureflix, Crackle, etc. Google, Facebook, Baidu, Amazon, the self driving car companies are doing cutting edge stuff in terms of scalability and analysis, but not Netflix. The complexity of their operations, seems mainly to be one of their own doing and not intrinsic to the service they provide. So I look at stuff like the article here and see a bunch of very smart engineers who are bored with the (solved) core problem and spend their time making cool stuff which is actually a pretty good thing.
- mdekkers 10y agoDon't they also have some kind of recommendation engine? I bet that takes a bit more data and processing (who watched what, did they finish, maybe even more like what kind of scenes did they stop watching, or watched again, etc)
- pkaye 10y agoI can see this kind of over engineering in all the companies that hit the VC/IPO lottery. They all hire like crazy the best engineers and have to keep them busy doing challenging engineering stuff. And managers want to protect their turf so they join in. So you have ever more complex designs in anticipation of growth, side projects, detailed analysis, blogs with pretty graphs, etc. A combination of Parkinson's Law and Second System Effect.
- gbrown_ 10y agoIs anyone else having issues accessing this? $ curl http://techblog.netflix.com/2017/04/the-evolution-of-container-usage-at.html <!DOCTYPE html> <html><head> <meta http-equiv="content-type" content="text/html; charset=UTF-8" /> <title>Access Denied</title> <style type="text/css">body {margin:0;font-family:verdana,sans-serif;} h1 {margin:0;padding:12px 25px;background-color:#343434;color:#ddd} p {margin:12px 25px;} strong {color:#E0042D;}</style> </head> <body> <h1>Access Denied</h1> <p> <strong>You are attempting to access a forbidden site.</strong><br/><br/> Consult your system administrator for details. </p> </body>
- thealfreds 10y agoWork filtering out netflix.com? Old place use to do it and it would bug me I couldn't read their techblog at work.
- gbrown_ 10y agoYup that appears to be it. Didn't even realise my workplace filtered anything.
- michaelper22 10y agoA previous workplace of mine does block Netflix, and consequently their tech blog.
- rodionos 10y agoI'm looking at Netflix repos at https://github.com/Netflix https://github.com/Netflix. 5 pages x 30, that's a lot of repos.
- bjornlouser 10y ago"We run a peak of 500 r3.8xl instances in support of our batch users. That represents 16,000 cores of compute with 120 TB of memory." I get that they are bragging about their implementation, but what number for peak batch processing instances would they be embarrassed to divulge?
- RangerScience 10y ago> In each of these examples, a key to the success of Titus was deciding what Titus would not do, leveraging the full value other infrastructure teams provide. This. So much this.
- johnsmith21006 10y agoWhat has happened to HN? This thread is so filled with misinformation it is clear there is very little understanding of container. A container is just a process. Really no different than any other. So cache and shared libraries, etc all the same with just a little care.
- lkrubner 10y agoThis is some amazing scale: ------------------- We run a peak of 500 r3.8xl instances in support of our batch users. That represents 16,000 cores of compute with 120 TB of memory. We also added support for GPUs as a resource type using p2.8xl instances to power deep learning with neural nets and mini-batch. In the early part of 2017, our stream-processing-as-a-service team decided to leverage Titus to enable simpler and faster cluster management for their Flink based system. This usage has resulted in over 10,000 service job containers that are long running and re-deployed as stream processing jobs are changed. These and other services use thousands of m4.4xl instances. While the above use cases are critical to our business, issues with these containers do not impact Netflix customers immediately. That has changed as Titus containers recently started running services that satisfy Netflix customer requests.