9 ms·
Adopting Microservices at Netflix
- matthewmacleod 12y agoNetflix doesn’t have an HR manual. There is a single guideline: “Act in NetFlix’s best interest.” The idea is that if an employee can’t figure out how to interpret the guideline in a given situation, he or she doesn’t have enough judgment to work there. If you don’t trust the judgment of the people on your team, you have to ask why you’re employing them. It’s true that you’ll have to fire people occasionally for violating the guideline. Overall, the high level of mutual trust among members of a team, and across the company as a whole, becomes a strong binding force. Christ on a bike, that sounds horrific. All this approach can do is replace codified guidelines with unknown, arbitrary ones. Who defines what "NetFlix's best interest" is? In an organisation of that size, conflicts of interest must happen all the time. Is this just a fancy way of saying "If we don't like you, you're out of the door?"
- baddox 12y ago> Is this just a fancy way of saying "If we don't like you, you're out of the door?" I'm sure it is, but is there any company where that isn't the policy?
- brendangregg 12y agoG'Day, I work at Netflix, and we hire what's been called "fully formed adults". Here's a quote that can explain it better than I (also see the full article): Source https://hbr.org/2014/01/how-netflix-reinvented-hr/ar/1 https://hbr.org/2014/01/how-netflix-reinvented-hr/ar/1: "Hire, Reward, and Tolerate Only Fully Formed Adults Over the years we learned that if we asked people to rely on logic and common sense instead of on formal policies, most of the time we would get better results, and at lower cost. If you’re careful to hire people who will put the company’s interests first, who understand and support the desire for a high-performance workplace, 97% of your employees will do the right thing. Most companies spend endless time and money writing and enforcing HR policies to deal with problems the other 3% might cause. Instead, we tried really hard to not hire those people, and we let them go if it turned out we’d made a hiring mistake." I trust myself, and everyone on my team to act in Netflix's best interest. Throughout my career, I've worked with many professional engineers who could have been trusted to work in the company's best interest. But there have been many times when we've been prevented to do so, due to process. So I've been fascinated at how Netflix is operating, and it's been great working here. It's important to know that a key role of management is to provide context to employees: what problems exist, what challenges we are facing, what opportunities might exist, what's important to Netflix right now. So that we know what to do, and how to exercise judgement. I've written about working at Netflix earlier this year on my blog (someone else posted a link).
- mbca 12y agoWhile this isn't nearly as bad as companies like Valve which go around claiming not even to have any managers, let alone any policies, it still means you're going to run into this: http://en.wikipedia.org/wiki/The_Tyranny_of_Structurelessness http://en.wikipedia.org/wiki/The_Tyranny_of_Structurelessnes...
- nkozyra 12y agoI think rather than "fully-formed adults," what's described here is mutual compatibility. Which is kind of unfortunate, as weird as that sounds. We'd all like to work with people who operate with the same understanding, ethos, sense of humor, etc. It makes the day better, right? No office drama, no awkward meetings. But I would contend doing this tends to put you at a genetic disadvantage, and that some types of confrontation are essential for growth. Introducing only like-minded people will produce linear results. What codified HR rules do is put some workplace guidelines around humanity - that allows a dissonant group of varied people to, you know, be themselves without necessarily worrying about conforming to avoid ruffling feathers. Perhaps it's more nuanced than that, but it sounds more like "hire people you like" expressed somewhat condescendingly as it is.
- Kalium 12y ago> What codified HR rules do is put some workplace guidelines around humanity - that allows a dissonant group of varied people to, you know, be themselves without necessarily worrying about conforming to avoid ruffling feathers. I hate to be the one who says this, but that's not what those guidelines do. They don't allow varied people to be themselves. They make everyone be a self-similar group of other people. Other people who are not themselves.
- brendangregg 12y agoFully-formed adults does not mean like-minded people, which is a different topic (groupthink). Expressing competing professional views is indeed healthy, and it's part of our culture deck: http://www.slideshare.net/reed2001/culture-1798664 http://www.slideshare.net/reed2001/culture-1798664 slide 13: "You challenge prevailing assumptions when warranted, and suggest better approaches" slide 14: "You say what you think even if it is controversial"
- caoilte 12y agoI don't disagree, but I thought this was an interesting tweet on the subject "The more talent density u have the less process you need. The more process u create the less talent you retain. -- Reed Hastings (@netflix)" https://twitter.com/billwscott/status/571761964839505920 https://twitter.com/billwscott/status/571761964839505920
- a_c_s 12y agoThe second sentence I agree with, but not the first. An 'untalented' developer is one who is perhaps slow to complete features, needs work on their design patterns, produces buggy code, etc. Process isn't needed for these people, rather it is for people who do or try to get away with things like: expense excessive purchases, frequently lie about being sick to get out of working, speak or act in a lewd or unprofessional manner, etc. Nobody is under the impression that mandating wearing a business suit (process) makes one suddenly better at programming (talent), the converse is equally absurd. edit: punctuation
- eddieroger 12y agoProcess isn't wearing a suit to work. Process is having to fill out paperwork and present to a change review board to do things like modify a DNS record or install a service. Process is to keep morons from breaking what everyone else is doing correctly. If you have a team of people who know how to manage DNS in a way that won't explode, the request process be pared down to an email asking them to do it and trusting they will, them trusting you're not asking for no reason, and management trusting it won't jeopardize the system.
- taeric 12y agoSimilar to Scotsmen, a true process(tm) is whatever successful people do to get work done.
- HillRat 12y agoTo be fair, the line about "Act in Netflix's best interest" seems to apply to expense account reporting (http://voiceglance.com/netflixs-anti-establishment-pro-cash-live-and-let-die-culture/ http://voiceglance.com/netflixs-anti-establishment-pro-cash-...), which seems like a reasonable rule. On the other hand, your basic critique rings true: everything I've heard about Netflix's internal culture -- or, to be precise, how Netflix likes to position its internal culture -- seems a little too self-consciously "high speed, low drag," to the point where even the laudatory article I linked to admits that "workers feel legitimately afraid of losing their jobs at all times." Yes, Netflix is full of amazing engineering challenges, and I'm sure compensation is excellent, but I doubt you can underestimate the psychological toll of working in an environment that makes stack-ranking look like a union shop.
- deleted 12y ago[deleted]
- louwrentius 12y agoI don't know how life is at Netflix. But I don't read it like that. I read it like I would want to work there in a heart beat. Because it seems that Netflix trust you and doesn't (micro)mange you on bullshit KPI's or something. It seems to me that at Netflix, you are trusted. You are trusted that your work improves the environment. Your work is evaluated afterwards. There is a risk of things going wrong, but with good, responsbile, knowledgable people, this is not so high. Part of this is that you take the initiative to set expectations and clearly discuss what you are planning to do and what you'll be working on so there is no misunderstanding. It's all about setting expectations and communication. I'm actually quite curious how life would be inside this culture. Since you have so much responsibility and control, life should be quite bussy, but manageable. The ideal would be that you just have a bunch of adult, professional people who manage themselves just by talking to each other, agreeing on how they want to work with each other to get things done and they manage / steer this process themselves just with open communication. No (micro)managers needed. The key is the people and their values / culture. Very simple things may make all the difference. Being on time. Being prepared for meetings (reading the short memo upfront, maybe lookup some things or make sure that you have done the tasks required of you). Starting the meeting on time. Respect each others time. A willingness to to share and help people understand how tings are setup and why (reasoning), document the stuff that is important. And you know what is important or are capable of just asking what is important for other people to know so what you document is relevant. I can only dream. http://www.brendangregg.com/blog/2015-01-20/working-at-netflix.html http://www.brendangregg.com/blog/2015-01-20/working-at-netfl...
- xj9 12y agoThis is precisely what I was picturing. If I can ever get my own business off of the ground, I hope to have it work similarly to what you described.
- harrylove 12y agoThis would seem to echo Hastings' Netflix culture slide deck published in 2009[1]. It's worth reading even if you disagree. The deck was the first time I had heard of a major company having a "no vacation tracking" policy. It was presented in the same context as the HR manual. The policy of having no policies is built on employee trust. Employee trust is built on the standard of only hiring top quality employees. I've never worked for Netflix or any other company willing to make such bold statements so can't speak to the reality. It would be interesting to read a first-hand account from someone who has been at Netflix since the time such non-policies were enacted. At the time, it sounded like a fantastic experiment. I would love to see the results. 1. http://www.slideshare.net/reed2001/culture-1798664 http://www.slideshare.net/reed2001/culture-1798664
- ryandvm 12y agoKind of a tangent, but having worked at two companies that have a "unlimited" vacation policies, I can say that I detest the concept. It's basically an anti-policy. There are two major ramifications of unlimited vacation policies: 1) It changes how you feel about taking time off. Rather than paid time off being something that you have "earned", you evaluate your PTO in comparison to your teammates - specifically most people try to make sure they are not taking off more than the average. The end result being everyone is subconsciously competing with everyone else to consume less PTO than average. 2) Since you have not accrued vacation time, your company does not owe you financial compensation for vacation time not spent when you leave. As someone that changes jobs every 3 or 4 years, this is more painful than it sounds. If I ever decide to be in a position that affords me the opportunity, I think I would enact a "minimum vacation policy". It's still unlimited, for whatever that's worth, but employees will be required to take a minimum amount of vacation every year.
- harrylove 12y agoSeems like #1 would be a symptom of having the policy without the culture to back it up. Was that your experience? For #2, I've worked for several companies, only one of which had a compensation package for unused vacation. All the rest have been use-it-or-lose-it by the end of each year. Getting paid for unused time sounds like a luxury. But I also recommend negotiating for more time off when you sign on, especially if you have several years of experience in the industry. Even if a company says first-year employees get X days of vacation you can still negotiate. Especially for cash-strapped companies, vacation may be the only part that can change.
- caw 12y agoThis other quote seem horrific as well. > A single specialist in Java distributed systems is managing the entire configuration without any commercial storage tools or help from engineers specializing in storage, SAN, or backup. So this person is either a genius who has specialized knowledge in these areas, or it's been abstracted by the platforms people to the point where it's only Java knowledge. Either way, it sounds like this person never gets to take a vacation or be not on-call. Hopefully there's not just 1 person responsible for all of the configuration, it just takes 1 person to run it. > Process prevents problems. At many companies, the standard response to something going wrong is to add a preventative step to the relevant procedure. The linked section goes on to talk about HR process with the "Act in NetFlix's best interest". It totally avoids the concept of technical processes. I'm betting "best interest" is a bit more formal on the tech side. Bad process is process for process's sake. Good process serves a very important role for offloading what people have to remember, as well as knowledge transfer. From time to time, steps of a process become irrelevant because of other changes and it has to be pruned. Sometimes companies forget about this step. If Netflix has ever seen a problem with their builds/deploys and added a test to check against that problem, guess what--they followed the standard response to update the procedure. It's not necessarily a bad thing.
- mpdehaan2 12y agoNot to speak for Netflix on the storage part, but Cassandra can run against ephemeral disks in AWS, and is designed for it. As such, I think what they are saying they don't have to think about storage as much.
- hibikir 12y agoYes, it is pretty arbitrary. It's not very different than how Valve works, other than valve firing and hiring less people. Now, what you have to take into consideration is that it's written as a contrast to all kinds of other places that run a command structure where it's your boss' job to figure out what is good for the company. I don't know about you, but I have worked in those environments before. They attract bad people, as doing what seems better for the company, as opposed to what is better for your boss, is discouraged. I think of one example in my past, where the team worked for two months on a feature we knew our in house customer did not want, and would refuse to use. However, it helped our boss' internal political status, so the team tolled, the feature was unused, and in the end, half the team quit. Our former manager, however, is still doing well, because what is good for the company was not what was good for his boss either. I often get recruiting calls from that company. They have trouble retaining good talent. Imagine that. So the Netflix approach brings in people that are more dedicated to the business than the chain of command. There's a lot of very good engineers there, and it's a pool that is often untapped outside of startups. It's not a bad strategic decision, IMHO.
- AngusMcQuarrie 12y agoInteresting how we're happy to talk about companies in a Positive light that do good things, but don't want to name names when it comes to the shit-holes we've ejected ourselves from.
- chc 12y agoIs that really interesting? I would be surprised if people were more eager to make enemies and potentially make themselves look petty than to make friends.
- q2 12y agoWe expect freedom and transparency in all aspects across the world irrespective of personal/social costs associated with it. Isn't it? So why exception in this case? Just because it is unwritten and some sort of convention inducing hypothetical fear? Won't it help future employees from joining such toxic work places? Just a view point.
- mgkimsal 12y agoThis seems like something where "when it works, it works great", but when it doesn't... I can easily see just a couple rotten apples spooling that who could damage that whole system very quickly.
- danudey 12y agoAnd the rotten apples get sacked, according to this post: https://news.ycombinator.com/item?id=9185110 https://news.ycombinator.com/item?id=9185110
- mooreds 12y agoWould love to hear from any Netflix employees lurking around here.
- djvu9 12y agoThis is similar to what people in those despicable countries are doing. In the name of "best interest for the country" people in power can do whatever they want. "Best interest" is hardly a measurable or even agreeable term.
- mc32 12y agoI agree this sounds on the surface like it's very liberal but in effect it's stifling. Since there are no concrete outlines the onus is on you to figure things out. That sounds great but then like you said, who decides "best interest"? It's an ideological wet dream. It's like saying, you can make any choice, you have the freedom, just make sure you make the "right choice". Not to mention what happens when netflix's best interest conflicts with outside interests.
- pnathan 12y agoThis is exactly what happens in the situations I've been in. It's miserable when you don't turn your attention to greasing the political game daily.
- RandallBrown 12y agoI work for a company with a similar HR Manual. It simply reads "Use good judgement." The company is Nordstrom, a retail store that has been around for more than 100 years and comes up often in "Best Places To Work" articles. It's pretty great.
- donpark 12y agoIf Netflix's goal is to attract more easy-going employees and avoid paranoid engineers, it's working for them.
- wonnage 12y agoThe logical conclusion of your codify-everything approach is this: http://en.wikipedia.org/wiki/Work-to-rule http://en.wikipedia.org/wiki/Work-to-rule Every functioning organization requires faith in the people you work with.
- ernestipark 12y agoI don't think so. This is similar to HubSpot's policy (and probably influenced) of "Use Good Judgement". You can read more about it in the culture deck[1], but it's incredibly liberating and means there is a culture of trust rather than rules and regulations. And in most places, employment is at-will anyways so outside of illegal reasons, you can get fired for any old reason anyways. [1] http://www.slideshare.net/HubSpot/the-hubspot-culture-code-creating-a-company-we-love http://www.slideshare.net/HubSpot/the-hubspot-culture-code-c...
- xena 12y agoThis looks like a thinly worded ad for Nginx Plus.
- look_lookatme 12y agoSure, it's called content marketing and there is nothing inherently bad about it.
- samspot 12y agoI think it is thickly worded. Entire sections have nothing to with the technology you choose. I particularly liked the culture section.
- fa 12y agoI don't understand the dichotomy between speed and efficiency. What's speed is the author talking about, and whose efficiency?
- acaloiar 12y ago"If you ask any developer whether a slower development process is better, no one ever says yes. Nor do management or customers ever complain that your development cycle is too fast for them." I've never known a developer not to speak up when the development cycle is too fast or a user who is not averse to the resultant buggy software.
- wahnfrieden 12y agoFaster dev cycles can be safe, even safer, as long as the changes are accordingly incremental. See the whole "continuous delivery" line of thought.
- kitbrennan 12y agoYou're confusing 'development cycle' and 'pushing code'. Pushing code too fast is clearly a problem that causes bugs, and neither developers or users are a fan of. The 'development cycle' includes the total time to implement something, including time taken to fix resultant bugs. Where you can speed up the development cycle without introducing more bugs (eg. by implementing comprehensive testing and QA practices), then no one is going to complain.
- acaloiar 12y agoI think you draw an important distinction, but I don't believe that either of our comments are incorrect. I was explicitly referring to a development cycle that is too fast. After all, developers are humans and not machines. Perpetually fast development cycles lead to mental exhaustion, missed vacations--bugs. Unless, of course, you exist on a team that is staffed over-capacity such that the absence of any single developer has an immaterial effect on development cycles. I suspect that is not the case for most of us.
- Consultant32452 12y agoI've had a customer complain that I was being too productive for their team of business/design folks to keep up. I literally got pulled aside and asked to slow down. I think it was making them look bad.
- tracker1 12y agoIt's interesting that this is an advertorial for nginx ... I'm currently working on implementing a microservices system, but have instead opted to use ZeroMQ for the connections, and utf-8/json for the request/response interfaces... Which in my mind seems a bit more sane once you reach behind the publicly accessible API (though also means that you have to wrap anything the public/website needs.
- nawitus 12y agoThe submission title should include 'Lessons for Team and Process Design', as it's a part of a series.
- pnommensen 12y agoI agree, the title was shortened after I shared the link here.
- ecesena 12y agoWhat I'm missing is actual examples of services split in micro services, with particular focus on the db (originally one db, then split in multiple dbs, one in each micro service). From the talks I kind of inferred that sharing is a good example of micro service, but that's also a good example of something that one could outsource to addthis, sharethis and similar (at least, I'm speaking from the perspective of a relatively young startup).