31 ms·
How We Build Code at Netflix
- mattiemass 11y agoVery cool article. Amazing how much tooling Netflix has built themselves.
- hitekker 11y agoI believe at least some parts of the tooling were developed by an embedded team from Pivotal Labs.
- bmoyles 11y agoNot that I am aware of. It's possible they contributed to some of our OSS efforts, but I don't believe anything used internally was developed in that manner.
- lintoma 11y agoPivotal ( the company that engulfed Pivotal Labs and Spring ) did contribute the Cloud Foundry integration in Spinnaker. This is not used at Netflix, though.
- cryowaffle 11y agoHuh? The article describes a massive system and there is a lot of abstraction here, are you suggesting Pivotal was a part of the large abstraction or just grabbing credit for some minor part?
- riskable 11y agoWhat's amazing is that Netflix let their teams develop solutions from scratch. Usually at big companies when their developers say something like, "can't we just build a solution ourselves?" they're laughed out of the room or, more likely, marked down as candidates in the next round of layoffs.
- chatmasta 11y agoIt helps that they have repeatedly proven this strategy works. When the CTO goes to the CEO and says "we're going to build this, not buy it," the CEO obviously trusts his decision making because he's seen him successfully execute his strategy so many times. It sounds like Netflix engineering culture is built a lot around trust. Management trusts that only top engineering talent works there, and the sink-or-swim culture, coupled with performance bonuses and a rising stock price, ensures that the engineers are making the best decisions for the company. It's a lot easier to approve a "build" vs "buy" decision when you know that the interests of the engineers pushing for it are actually aligned with the company business interests. Contrast this to a company with a large separation of incentives between the stock price and the engineers. In that scenario you end up with a bunch of engineers who are bored and want to prematurely optimize systems because there is no clear personal cost to doing so.
- riskable 11y ago> It helps that they have repeatedly proven this strategy works. I laughed at this because you can't "repeatedly prove" you're capable of developing your own solutions if you're never allowed to do it in the first place! The enterprise mantra is: COTS or it doesn't happen.
- jdblair 11y agoFairly new Netflix employee here (started last November). From what I can see so far the "highly aligned, loosely coupled" culture here works very well. The company has a clear focus, we all get it, and we all work to make it happen. Teams have freedom to accomplish their goals they best way they see fit. There is quite simply not a lot of focus on particular means, the focus is on outcomes.
- draw_down 11y agoI've been in so many places where that response would be the correct one. I've spent a lot of time decommissioning some rather horrible outcomes of "can't we just build it ourselves".
- moondev 11y agoSpinnaker is an amazing tool. Really makes it easy to confidently deploy applications via immutable infrastructure.
- sanjeetsuhag 11y agoCan some one explain to me what immutable means in terms of "infrastructure" ?
- davis_m 11y agoIt means updating infrastructure by making changes to a versioned service definition instead of on running instances. Would you fix a software bug by editing the code on a running server and tell yourself that you will add it to the repository later? Of course not. You would end up with a running instance of the code that is impossible to replicate. Immutable infrastructure applies that same idea to running services.
- seiji 11y agoWould you fix a software bug by editing the code on a running server and tell yourself that you will add it to the repository later? Absolutely yes in the right circumstances (have you seen how this website works?). Not every service needs to be written as "scale out to a billion nodes" architecture with eight layers of checks and balances between idea to production. We can't globally say "everybody must use a 16 step verifiable app development pipeline" when some people run 3 servers and others run 3 million. (Plus, not all services are stateless! Redeploying stateful servers is painful—you'll be killing user experience. Updating code live on a running server while maintaining internal continuity of state can maintain sessions/caches/game-state without annoying users by kicking out of their current flows.)
- wpietri 11y agoI agree with parts of this, sort of. I maintain a personal server that hosts a variety of VMs for things like mail, static site hosting, nameservice, etc. Last time I rebuilt it, I thought, hey, I'll try doing it with these interesting tools. The immutable-infrastructure approach turns out to be pretty heavy in that case. A lot of times it feels like I have to perform a triple bank shot just to make a minor configuration change. That said, I think the one-off server, carefully maintained by one person, is becoming very rare. When I build things that other people will work on, I think immutable infrastructure and the cattle-not-pets approach is the only responsible way to go. Even if traffic volume won't be huge, I think the clarity and ease of debugging you get is vital when it doesn't all live in one wizard's head. I agree that stateful servers are a challenge in this context. But they're a challenge regardless. Having that one thing we're afraid to upgrade or restart can cause all sorts of development and business process issues. That might have been a worthwhile tradeoff 10 years ago when you had to buy all your hardware and you needed ops people to do a lot of stuff manually. Servers go down, and we might as well accept that from the beginning.
- pitt1980 11y agobm
- dang 11y agoPlease stop posting these.
- pitt1980 11y agowhy? I'm trying to keep track of things that catch my eye
- ones_and_zeros 11y agoif you upvote something it'll show up in the list at https://news.ycombinator.com/saved?id=pitt1980 https://news.ycombinator.com/saved?id=pitt1980. So just upvote things that catch your eye.
- TallGuyShort 11y agoYou can upvote a story an it will appear under "saved stories" when you click on your profile. You commenting to save stories for you is spamming everyone else. So yes, please stop.
- pitt1980 11y agothanks, I hadn't noticed that feature before
- dang 11y agoIt's an abuse of HN threads. Comments here are supposed to be substantive.
- loco5niner 11y agoSure, it's useful to you, but it adds to the clutter that hundreds and thousands of others must wade through.
- NeutronBoy 11y ago
- Scarbutt 11y agoWhat are the reasons for Netflix choosing nodejs for their front-end server and not java like in their back-end?
- salehenrahman 11y agoI'm guessing it's their use of React both on the front-end and back-end.
- dominotw 11y agothere is also this https://github.com/Netflix/falcor https://github.com/Netflix/falcor
- aaronblohowiak 11y agoFreedom and Responsibility! Teams are free to choose the tools that work for them. Netflix technical direction operates more like an ecosystem than a hierarchy.
- error54 11y agoSince all Frontend code is written in JavaScript, it makes sense to have the server code written in JavaScript as well as we do quite a bit of work on the server. Also, we do serve isomorphic views so Node is a big help in that regards.
- btbuildem 11y agoBecause using Java for a front end server is a horribly inefficient and painful experience.
- agentgt 11y agoJava even with webjars, hot reloading (jrebel), and what not just can't keep up with the superior Node tooling not to mention more and more frontend developers know node and not Java. I still prefer Java/JVM languages though for backend work.
- oconnore 11y agoAnother Spinnaker: http://arxiv.org/pdf/1103.2408.pdf http://arxiv.org/pdf/1103.2408.pdf
- simonebrunozzi 11y agoGood point - curious what do you exactly mean by this though, and how did you compare Spinnaker with it. Can you elaborate a bit more?
- oconnore 11y agoReally just a naming collision. IBM Spinnaker is a data store, Netflix Spinnaker is a software builder. The paper I linked is interesting if you're researching distributed data stores.
- vlucas 11y agoFor those wondering about how this applies to Node.js use a Netflix like myself, it's in there towards the bottom of the article: > "As Netflix grows and evolves, there is an increasing demand for our build and deploy toolset to provide first-class support for non-JVM languages, like JavaScript/Node.js, Python, Ruby and Go. Our current recommendation for non-JVM applications is to use the Nebula ospackage plugin to produce a Debian package for baking, leaving the build and test pieces to the engineers and the platform’s preferred tooling. While this solves the needs of teams today, we are expanding our tools to be language agnostic."
- agentgt 11y agoTwitter has an agnostic tool called Pants [1]. I run a JVM shop and we are just now using some NodeJS so we are have been trying to figure out that as well. [1]: https://pantsbuild.github.io/dev.html https://pantsbuild.github.io/dev.html
- neduma 11y agoHow do they 'externalize config' with respect to http://12factor.net/config http://12factor.net/config?
- conorgil145 11y agoThere are many ways to accomplish an external configuration. I do not know how they do it at Netflix, but one could have the prebaked AMI reach out to a service discovery tool to pull its configuration at run time. For example, Consul [1] or etcd [2]. One advantage of this approach is that you can use the exact same AMI in testing as in production. All you would have to do is change the configuration which is pulled. You could determine which configuration to pull based on AWS tags on the EC2 instance, for example. [1] https://www.consul.io/ https://www.consul.io/ [2] https://github.com/coreos/etcd https://github.com/coreos/etcd
- bmoyles 11y agoA combination of things such as Spinnaker providing granular environmental context to an instance by way of user data (things that the ec2 metadata service doesn't readily provide to an instance without API calls), dynamic properties which can be controlled down to the instance-level (http://techblog.netflix.com/2012/06/annoucing-archaius-dynamic-properties.html http://techblog.netflix.com/2012/06/annoucing-archaius-dynam... ), and other solutions depending on a team's needs.
- markbnj 11y agoI'm a Netflix fan, as a consumer and an engineer, and this blog post just reinforces my fanboi status. Amidst the descriptions of deployment tools and pipelines one thing stood out for me: the fact that AMI bake times are now a large factor, and that "installing packages" and the "snapshotting process" were a big piece of this. Containers are definitely the answer to this problem. You can deploy base images with the OS and common dependencies, and have the code changes be a thin final layer. Of course with such a sophisticated pipeline based on AMI deployment this change would not be trivial for Netflix, but the bottom line is they have described the primary container use case perfectly, imo.
- mkobit 11y agoI am fairly certain it is on the Spinnaker roadmap. There was a recent talk by Sangeeta Narayanan on "Containers so Far" [1]. At 37:10~ you can see how they will essentially be allowing for containers to replace AMIs in the "bake" process. I'm looking forward to seeing how Netflix start to roll out more container support, and seeing how Spinnaker progresses. [1] http://www.infoq.com/presentations/netflix-containers http://www.infoq.com/presentations/netflix-containers
- Hovertruck 11y agoAt Button we switched from baking AMIs to deploying containers on ECS and it's had a pretty amazing effect on deploys, both in terms of the speed and (due to them being quick and painless) frequency.
- sayrer 11y agoSeems like a nice system, but would be improved by building with Bazel or Buck instead of Gradle.
- mkobit 11y agoI'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do everything" workflow system. The easy path that Jenkins provides, to me, seems like the wrong one - it makes it easy to just stuff everything in there because it "can" do it. This seems to leads to tons of copy/paste, drift, all types of different work being represented, and it is starting to become unmanageable. Have others seen this happen when using Jenkins? How have you dealt with it?
- richardwhiuk 11y agoYes - we've definitely seen that. The most common thing we've see if that you start with a jenkins job that just runs a build process, and then you end up with a jenkins job that has a huge shell script which calls the build process somewhere in the middle, that's unversioned. To resolve that we've tried to push the shell script into a file in the repository, which Jenkins then checks out and runs, which makes it easier to maintain and faster to set up new build machines.
- christop 11y agoNetflix have been quite involved in the Jenkins project, including the Job DSL Plugin, which enables the automated creation of new Jenkins jobs, e.g. when a new Git branch is created, by defining the job structure with a simple Groovy-based DSL. Taking this further, the upcoming release of Jenkins 2.0 is going to put a lot more emphasis on pipelines-as-code, where entire workflows can be defined in code, and version-controlled, as opposed to clicking everything together via the web UI. See https://jenkins-ci.org/2.0/ https://jenkins-ci.org/2.0/
- oblio 11y agoSome things are already available, see this: https://github.com/jenkinsci/workflow-plugin/blob/master/README.md#introduction https://github.com/jenkinsci/workflow-plugin/blob/master/REA...
- x0rg 11y agoIsn't Netflix using Mesos (see http://techblog.netflix.com/2015/08/fenzo-oss-scheduler-for-apache-mesos.html http://techblog.netflix.com/2015/08/fenzo-oss-scheduler-for-... )? I don't understand what role it plays here.
- diab0lic 11y agoThe Fenzo scheduler is, if I recall correctly, used to do scheduling for Mantis and Titus (formerly Titan). Think of these as a streaming job infrastructure and a container service respectively. Spinnaker can assist you in deployments to both (I use Spinnaker to deploy Mantis jobs, for example) but the two platforms are not completely prolific -- we still deploy much of the ecosystem as baked AMIs.
- Gratsby 11y ago> The Netflix culture of freedom and responsibility empowers engineers to craft solutions using whatever tools they feel are best suited to the task. I absolutely love that. I'm a huge fan of what Hastings and company have done over there in terms of culture and making Netflix a unique and desirable place to work. I think it's time for another round of "find a way to make Netflix hire me."
- billjings 11y agoThe real interesting thing to me is the language around "paved road"s, and management support for tools teams to maintain those paved roads. It's easy to say, "Engineers should use the tools they deem appropriate" and make the engineers happy. What's harder is to support those individual efforts with attention and money to build a common consensus and a shared toolchain. "Paved roads" is nice language to describe the utility of consensus without condemning those who depart from it. I dig it.
- seanp2k2 11y agoThere's an open house coming up. Let me know if you'd like to come (if you're in the bay area) zgtjyizv@abyssmail.com
- Gratsby 11y agoAwesome. I just sent you a message. Thank you!
- gjkood 11y agoMajor outage being reported worldwide. http://downdetector.com/status/netflix http://downdetector.com/status/netflix Anything interesting deployed in the last hour? Something in the CI/CD tool chain, Spinnaker, failed for it to move all the way to Live without being caught.
- cm2187 11y agocan't connect from the UK...
- codingdave 11y agoThere are many reasons that a site can go down, aside from a code deployment.
- relix42 11y agoNot deployment or spinnaker related
- pyman 11y agoThis is how my company used to build software 10 years ago. Now we have Docker containers, cloud VMs, GitHub, 1 click deployment, advanced metrics, Grafana, Go microservices, Slack bots, etc. Sounds like Netflix is stucked in the past.