10 ms·
A View of the Entire Netflix Stack
- orestes910 11y agoI really like that Netflix is constantly open about how their systems are run. In terms of raw throughput not a lot of companies can hold a candle to them, so its cool to see how the big players do things.
- XorNot 11y agoI would wager because, looking at that technology stack, most of it is commercial paid services? "We use EC2" just pushes the question to someone else.
- sciurus 11y agoThis isn't a very good article. Judging from the first table, netflix is written in node.js using python and java. :-/
- mxpxrocks10 11y agoThanks for the feedback. We heard this a little bit through twitter. Can you reference any links? Happy to make the change. It was a little tricky to put together with out interviewing someone. We want it to be accurate and useful. Please post any reference links here and we'll follow up with an edit.
- mxpxrocks10 11y agofor example, there was reference to the Spring framework - any reference links are appreciated.
- mschulze 11y agoSpring even integrated some OSS projects from Netflix into Spring Cloud: https://github.com/spring-cloud/spring-cloud-netflix https://github.com/spring-cloud/spring-cloud-netflix
- andrew_mason1 11y agoWell, node.js implies that at least part of Netflix is written in JavaScript, yet the only listed languages are Python and Java.
- mxpxrocks10 11y agoFair enough. Will modify shortly and good point.
- mxpxrocks10 11y agoOK - updated it on scalescale.com (the original post) and gave you credit in the comments. Good find guys. Appreciate it.
- zappo2938 11y agoSomeone at Netflix did a talk perhaps it was about Falcor. They use React embedded in browser apps across all devices. So they use Javascript on the front end. They use Node.js as the proxy server / router with a framework called Falcor which is written in Javascript. Falcor queries all the multitudes of services and databases then caches that data in a graph. Falcor reaches out to all the servers and databases, scores or services, that are written in Python and Java.
- mikeryan 11y agoPicking a nit, they don't use React in the browser across all devices. They actually use an internal rendering engine called Gibbon on some devices with client code written JavaScript with React but not rendering to HTML or a browser instead using their internal engine. Interestingly this has become easier with React breaking out React and ReactDOM. Interesting to note that Netflix must see Gibbon as a competitive advantage, I've been told its not likely to see an open source release. https://www.youtube.com/watch?v=eNC0mRYGWgc https://www.youtube.com/watch?v=eNC0mRYGWgc
- jedberg 11y ago> It was a little tricky to put together with out interviewing someone. Why didn't you just interview someone then? There are plenty of folks there who would happily grant you an interview. If I still worked there I would have.
- mxpxrocks10 11y agoright on. PS cloudnative.io looks cool! Looking forward to playing around with it.
- efuquen 11y agoI know they use Scala: https://skillsmatter.com/skillscasts/5424-netflix-scala https://skillsmatter.com/skillscasts/5424-netflix-scala http://hanselminutes.com/394/demystifying-scala-with-netflixs-dianne-marsh http://hanselminutes.com/394/demystifying-scala-with-netflix... https://github.com/Netflix/atlas https://github.com/Netflix/atlas
- mxpxrocks10 11y agothanks for the links! Atlas is on the post.
- deleted 11y ago[deleted]
- monksy 11y ago> Sustained B-level performance, despite "A for effort", generates a generous severance package, with respect. You've got to love these startups who basically say "if you're not giving us everything beyond what we expect... go screw yourself. I'm not sure how Netflix grades things... but in school B is "Above average". Is Netflix looking to employee and develop people or are they looking to extract life from people?
- deleted 11y ago[deleted]
- lechevalierd3on 11y agoSounds like the right way to make sure you only attract great team players.
- monksy 11y agoThere are really a few outcomes on that: you're going to hire people that will rip each other apart to get the A ranking, your going to get a group of people that are so afraid of screwing up that they're basically neutered, out your going to get everyone cooking the books. I can't see this requirement on the individual actually doing anything for the team.
- sbov 11y agoThe difference is that they back that statement up with A-level compensation. Most startups don't. Also, is Netflix still a startup?
- lappa 11y agoThis is a bit confusing. I don't suspect Netflix is built using primarily node.js, and I don't see how the Netflix Prize (algorithm for predicting what a person is most likely to want to watch) falls under "scaling".
- johansch 11y agoThis is nitpicking.
- lappa 11y agoAmong the things I had knowledge of, a lot seemed incorrect, so I wasn't quite sure how to take the things I didn't have knowledge of.
- such_a_casual 11y agoHonestly, there are enough problems with the article to warrant a critique. However, someone may not want to take the time to outline every single one. It's incredibly confusing to the reader that the article states Netflix relies on frameworks for one language, and then states that all of their code is written in another language.
- mxpxrocks10 11y agoHey, we got this fixed up on the original post- waiting for Todd to fix the repost on HighScalability shortly. If there are some more problems let us know. We want it to be accurate. http://www.scalescale.com/the-stack-behind-netflix-scaling/ http://www.scalescale.com/the-stack-behind-netflix-scaling/
- mxpxrocks10 11y agohey, thanks for the feedback lappa. We've updated the original article to display node.js better and added a comment. The idea behind mentioning the Netflix prize is to show the openness and willingness to get a lift on their existing recommendation algorithm. There was a 10+% lift in accuracy. Their ability to open up the data and implement really gives them a competitive edge IMHO. It's a shame there seems to be some regulatory reasons they don't do it any more.
- CodeSheikh 11y agoI am a bit surprised. Why host Netflix application on Amazon EC2 and not have your own data centers? A startup hosting at Amazon EC2 makes sense
- rpedela 11y agoWhy Netflix chose AWS: http://techblog.netflix.com/2010/12/four-reasons-we-choose-amazons-cloud-as.html http://techblog.netflix.com/2010/12/four-reasons-we-choose-a... Also Quora answer: https://www.quora.com/Why-does-Netflix-use-Amazon-EC2 https://www.quora.com/Why-does-Netflix-use-Amazon-EC2
- mentos 11y agoThe reason that strikes me as most important is #2 "Letting Amazon focus on data center infrastructure allows our engineers to focus on building and improving our business." Too many companies roll their own solutions when they could take advantage of economies of scale like this and let another company dominate the problem. Obviously there are privacy reasons to own your own data but I don't think they outweigh the performance and efficiency implications.
- qaq 11y agoaws and performance :) Crappy performance and poor uptime is precisely why people move off aws.
- jadavis 11y agoSo, um, why does Netflix always seem to be up/performant?
- njharman 11y agoBecause they actually know how to implement fault tolerant scale on AWS. Most companies don't have simian army randomly wrecking infrastructure. Many companies/engineers think they can spin up a few (or few hundred) ec2 instances and think they have solved resiliency. AWS is a tool, not a bullet made of silver.
- johansch 11y agoIs this reliance on a variety of external services why Netflix fails so often compared to e.g. youtube.com? I assume so.
- mxpxrocks10 11y agohttp://mashable.com/2015/11/10/netflix-site-goes-down/ http://mashable.com/2015/11/10/netflix-site-goes-down/
- kylek 11y agoNot sure if it was left out on purpose, but I'd love to see more details on their transcoding stack (they only mention that it is done in EC2)
- mxpxrocks10 11y agoHi Kylek - thanks for the feedback. It was a little hard on first try to find information on this. The only thing that seemed clear was that adaptive bit rate was done on the edge and transcoding was done and stored. Feel free to post links here if anyone has any.
- faizshah 11y agoAdrian Cockcroft gave a much better overview of their open source projects here: https://www.youtube.com/watch?v=R2kKmMyqTfc https://www.youtube.com/watch?v=R2kKmMyqTfc It's two hours long but the first hour is sufficient to get an overview of their open source projects.
- mxpxrocks10 11y agoGood call. Thanks for posting. We'll add this to the comments of the original article.
- mxpxrocks10 11y agofaizshah - posted (and gave you credit) on the original ScaleScale post. Thanks. BinaryRage - The link is there to netflix.github.io on the original post.
- BinaryRage 11y agoYou might want to link to the OSS site too - https://netflix.github.io/ https://netflix.github.io/
- fitzwatermellow 11y agoThis AWS re:Invent 2015 talk by Dave Hahn is also really illuminating: A Day in the Life of a Netflix Engineer (Using 37% of the Internet) https://www.youtube.com/watch?v=-mL3zT1iIKw https://www.youtube.com/watch?v=-mL3zT1iIKw Interesting to me that they use Atlassian Confluence for team stuff. Thinking of testing it out as well...
- swsieber 11y agoConfluence is pretty cool, but they suck at bullet lists and indenting code blocks... so basically they suck at a couple of basic UI stuff. Apart from that it seems pretty great.
- georgebarnett 11y agoAre you aware of the {code} macro which formats code?
- swsieber 11y agoI am. It does not indent properly though - there is only one level of stable code indentation (the entire block that is; it is possible to indent the code within the block, but I'd like to indent the code frame).
- mxpxrocks10 11y agogreat link! thanks for posting fitzwatermellow. Adding to the ScaleScale original post and giving you credit.
- such_a_casual 11y agoA word to the author: Please proof read your writing.
- mnml_ 11y agoIt must be really expensive tho (even with discounts). But I guess they don't care.
- steven2012 11y agoAs the comment at the end of the article mentioned, they forgot to add Spring. I've met a few Netflix engineers, and they said their Java implementation is done using Spring.
- mxpxrocks10 11y agohey Steven! Thanks. Do you have any links that reference this? We want it to be as accurate as possible.
- steven2012 11y agoNo links, I got this from actually talking with some engineers. Have you not contacted a single Netflix engineer to verify your article? If not, I find that bizarre.
- jacques_chester 11y agoIndeed the love has been returned. The Spring team have been wrapping and integrating a lot of Netflix OSS into Spring Cloud: http://projects.spring.io/spring-cloud/ http://projects.spring.io/spring-cloud/ http://cloud.spring.io/spring-cloud-netflix/ http://cloud.spring.io/spring-cloud-netflix/
- Randgalt 11y agoNo, it's not. It's primarily Google Guice via Governator.
- rufugee 11y agoOver the years, I've heard a few podcasts stating that Netflix relies heavily on Grails/Groovy for infrastructure management tools (https://github.com/Netflix/asgard https://github.com/Netflix/asgard, etc). Wonder why that isn't mentioned?
- mxpxrocks10 11y agoGood call - we'll dive deeper into it and add it.
- vorg 11y agoThey do say: "We are sharing the things we think are most interesting." Maybe a minor use of Groovy and Grails is only interesting to people trying to promote those products.
- clarle 11y agoOnly the UI layer of Netflix is in Node. Most of the cloud services are JVM-based (as is much of the NetflixOSS ecosystem [1]), and then the Node services connect to the JVM-based services via Prana. [2] [1]: http://netflix.github.io/ http://netflix.github.io/ [2]: http://ispyker.blogspot.com/2015/10/towards-being-better-about-open-source.html http://ispyker.blogspot.com/2015/10/towards-being-better-abo...
- chatmasta 11y agoFunny timing given that Netflix is currently down... [0] [0] http://www.isitdownrightnow.com/netflix.com.html http://www.isitdownrightnow.com/netflix.com.html
- morgante 11y ago@monsky: (Somehow this got disconnected from the thread below.) The headline of the slide you pulled that from is literally "Hard Work—Not Relevant." Netflix doesn't care if you work 100 hours a week, they care how well you perform. If you can do an amazing job in 30 hours a week, they're one of the few places which would actually embrace you. > I'm not sure how Netflix grades things... but in school B is "Above average". So what? Why on Earth should Netflix be targeting "average" workers when they deliberately pay twice the average. That's about as ridiculous as saying that Harvard should stop admitting A students because "B" is average. Netflix has a deliberate strategy of keeping their engineering bar very high and paying accordingly. That doesn't mean they want you to sacrifice your life to the company. Also, Netflix is in no way a startup.
- deleted 11y ago[deleted]
- erikb 11y agoThe problem about the idea of "doing great things in 30 hours" is that people who are not directly involved with implementation and deployment usually think that each and every 2 day task takes at most 2 hours, because we humans are unable to expect the complexity that most tasks have on this planet. (every time you think about a college, super market employee etc "just do your job man" you are experiencing that fallacy) And then you as the 30 hours employee are faced with not delivering 40 hours of work in 30 hours (which is possible if you are smarter, more involved, have more experience etc) but no you are expected to do 160 hours of work in 30 hours, because you don't compete with the other engineers but you compete with the expectation of your manager. We need to accept that most people (colleagues, employees, customers, bosses) will always be disappointed by the results we achieved because they had unrealistic expectations to begin with. And we must excuse them because we do the same the other way around. That is why "40 hour week" is so relevant. Because then you can say after working hard for 40 hours, really doing all you can, maybe even achieving 30% more than your colleagues, that time is up and you need to take care of your children, wife, parents, friends and hobbies you may have. Now let's assume for a moment that all I said was BS. Let's assume I'm super untalented and for most people it's actually possible to achieve in 30 hours enough to impress their leaders. Now you do it once. You do it twice. Maybe thrice. But at that point your leader will adjust in his head, maybe without realising, what 100% means for you. So the fourth time he doesn't think it's awesome anymore. The tenth time he thinks you are lazy. And now you have to achieve 50 hours of work in 30 hours. It still doesn't work out.
- brown9-2 11y agoSource for the bit on Mesos?
- mxpxrocks10 11y agocheck: http://techblog.netflix.com/2015/08/fenzo-oss-scheduler-for-apache-mesos.html http://techblog.netflix.com/2015/08/fenzo-oss-scheduler-for-... and post anything else
- bricss 11y agoOMG! Node.js in frameworks folks xD
- yarou 11y agoThis reads less like a technical overview of Netflix's stack and more like an advertisement for working there. I'll give you my two cents on the matter. Netflix spins a great narrative and ideal -- the rose-colored "meritocratic" dream of Silicon Valley. I'd suspect the reality of working there is a lot more mundane than the beautiful narrative that they (and by extension, the author of the post) are trying to sell you. There are probably a wide array of experiences that current employees have, ranging from pure joy to misery, which are contingent upon team, co-workers, management, compensation, etc. An interesting read nonetheless.
- gopowerranger 11y agoThe Netflix Open Connect CDN is provided for larger ISPs that have over 100,000 subscribers. A specially built low power high storage density appliance caches Netflix content within the ISPs’ data centers to reduce internet transit costs. This appliance runs the FreeBSD operating system ...
- Geee 11y agoWhy there's so much brag about Netflix while they can't fix even simple bugs and UX issues? After all, it's one of the simplest apps there are. Whole stack could be handled by one guy.
- chambo622 11y agoCurious what you mean by this, as it's overall one of the more polished products I use on a regular basis. Hasn't always been the case, and the Android app still does some weird UI things, but generally the product is solid and a pleasure to use. "Whole stack could be handled by one guy" - comments like like this and similar others ("I could build Twitter in a weekend", etc.) strike me as unproductive.
- Geee 11y agoFor example, Netflix consistently forgets what I have seen and if I don't watch the ending credits it thinks I haven't watched an item and suggests it in the continue watching section. There should be visual indicator on what I have watched, and if there are unwatched episodes in series. Now I have to open it and go to the episode list, which is unreliable. Also, starting video playback is too slow. It's not possible to open items in new tabs. There are no keyboard controls.
- smadge 11y ago> Whole stack could be handled by one guy. wat
- zobzu 11y agoThe A (promoted) vs B grade (fired) perf means: - No "regularly well performing" employees are kept. You're either promoted or fired. Ensure workforce moves. - Doing the shitty necessary work such as fixing tech-debt just gets you fired. But I bet people do it anyway, else Netflix would be in troubles. It's just that they get fired after that. Not really nice. It also means you've to be only doing stuff that has direct visible impact, and refuse any work that does not/place yourself visually above others. Not saying this is a bad strategy to get stuff to work, not sure if very ethical though. That's all I noticed was actually interesting ;-)
- kyllo 11y agoExactly--in a work environment when everyone is fearing for their livelihood, succeeding at performance theatre becomes the primary concern. You need to be a top performer, in both senses of the word--but it's human nature that over time the acting sense will take priority.
- Randgalt 11y agoThis view is missing some things. For example, Curator (now at Apache).
- mxpxrocks10 11y agohttp://curator.apache.org/ http://curator.apache.org/ "Guava is to Java what Curator is to ZooKeeper" Patrick Hunt, ZooKeeper committer
- JupiterMoon 11y agoSo which part of this stack is to blame for their slow clunky website? (NB their website used to work well enough on my devices the newer one is the problem.)
- insulanian 11y agoWhere is the Chaos Monkey? (http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html http://techblog.netflix.com/2012/07/chaos-monkey-released-in...)
- mxpxrocks10 11y agoSimian Army Suite https://github.com/Netflix/SimianArmy https://github.com/Netflix/SimianArmy