4 ms·

Why is DevOps is so hard in other companies yet so easy in Netflix? I mean, the entire monitoring stack of Netflix was automated by one person and managed by fo

Why is DevOps is so hard in other companies yet so easy in Netflix? I mean, the entire monitoring stack of Netflix was automated by one person and managed by four. Two people in Netflix used to manage their entire messaging pipeline, including suro, Kafka, Zookeeper, and some Druid stuff. Their Hadoop ecosystem used to have a handful of people and their end-to-end ingestion pipeline that aggregates and demuxes data into Hive with 15-minutes of maximum delay was written and maintained by a single person. Their key manager, well before Amazon KMS existed, was implemented and maintained by a single person. Their engineers in the cloud platform team were oncall 24x7 yet they only occasionally got paged. Oh, their cloud platform team had fewer than 20 people too. Their entire Cassandra team used to have fewer than 20 people. They predictively autoscaled their clusters with a system created and maintained by merely 2 or 3 people. According to their engineers, they did a few things on top of the excellent foundation of AWS, and they just did them without any fuss: - API-based full transparency. If there's a function, there is an API. Anything you can do is explicit in API. - Powerful monitoring support so "instrumentation to death" is not just a slogan. - Decentralized control. For instance, teach team has freedom to decide how to load balance their traffic and how to handle unresponsive nodes. - Assuming everything can fail and build mechanisms to account for that assumption. Chaos engineering. Autoscaling from get go (again, thanks to Amazon for such a powerful feature from day one in EC2). - A shared culture: don't tell me to learn your shit. So, no friction from handling shit like Puppet, like Chef, like Terraform, like HCL, like whatever yaml or DSL that folks on HN passionately advocate. Like learning how to embed a jinja template in a job description with 5000 lines of Ruby code plus some system-specific half-assed DSL just to update an environment variable? Never gonna happen. Don't get me wrong: they are probably nice technology. They are just too damn low level and irrelevant to most engineers. - Instant gratification. If I make a config change, I want to see it in production in seconds, safely. If I make a change in my code? The change will deploy with all the guardrail in seconds. So, a puppet change that takes on average 15 minutes to materialize? That's just garbage. - No surprise. So, a Chef script goes behind my back to update my OS and screws up my production service? Never happens. Immutable infrastructure was implemented from day one. So the question is, why are those thing hard in other companies? Do engineers enjoy getting paged at 2:00am? Do they enjoy spending at least 1/3 of their time handling so-called operations? Or do they enjoy writing rants like DevOps is a failure?

xyzzy123 4y ago
I don't get how 1 person maintaining key management service at Netflix works, I suppose there is something obvious I don't understand. In most organisations having 1 person fully manage / own a thing (in the usual sense of that) is a serious problem because that person needs to take holidays and can quit. Depending on type of org it can also be a security issue (e.g. you are likely to have a hard time if the service is in PCI scope). Usually you'd want at 4 people to be able to sustain a service reliably through seasonal flus and to cope with natural organisational attrition. At least 2 fully up to speed (so they can review/approve each other's MRs/PRs) and 2 shadows who have access to everything required and can be hands on. Anyway, how did that work?
- hintymad 4y ago
  Yeah, bus factor is real. I'm sure they later hired more people to manage the service. The point was that one person was able to manage the service without worrying about operations. The service does not go down. People provision keys and certs via an API so business continues even if the person went on vacation -- the only impact was that no one would be there to add new features.
  - xyzzy123 4y ago
    Yeah I think I'm aligned with you. So many processes at my workplace are pointlessly "service now" driven and would be enormously more efficient if they were provided by API. I think a lot of legacy organisations are held back by having large departments of "non-coders" in infrastructure roles. It's a sticky local minimum. You can't automate without enough code-literates around to support things. But the non-coders tend to hire more people like themselves. They are swamped with their existing problems and believe more of the same to be the solution. So things don't get automated...
Too 4y ago
It’s hard because far from everybody is Netflix scale. A monitoring stack maintained by five dedicated engineers is factor 10-100x more than any other small or medium sized company can afford to throw at it. Most places you’d be lucky to find even one engineer. One engineer working full time with only a specific tool is a luxury and one such engineer will be able to achieve a lot.

xyzzy123 4y ago

I don't get how 1 person maintaining key management service at Netflix works, I suppose there is something obvious I don't understand. In most organisations having 1 person fully manage / own a thing (in the usual sense of that) is a serious problem because that person needs to take holidays and can quit. Depending on type of org it can also be a security issue (e.g. you are likely to have a hard time if the service is in PCI scope). Usually you'd want at 4 people to be able to sustain a service reliably through seasonal flus and to cope with natural organisational attrition. At least 2 fully up to speed (so they can review/approve each other's MRs/PRs) and 2 shadows who have access to everything required and can be hands on. Anyway, how did that work?

hintymad 4y ago
Yeah, bus factor is real. I'm sure they later hired more people to manage the service. The point was that one person was able to manage the service without worrying about operations. The service does not go down. People provision keys and certs via an API so business continues even if the person went on vacation -- the only impact was that no one would be there to add new features.
- xyzzy123 4y ago
  Yeah I think I'm aligned with you. So many processes at my workplace are pointlessly "service now" driven and would be enormously more efficient if they were provided by API. I think a lot of legacy organisations are held back by having large departments of "non-coders" in infrastructure roles. It's a sticky local minimum. You can't automate without enough code-literates around to support things. But the non-coders tend to hire more people like themselves. They are swamped with their existing problems and believe more of the same to be the solution. So things don't get automated...

Too 4y ago

It’s hard because far from everybody is Netflix scale. A monitoring stack maintained by five dedicated engineers is factor 10-100x more than any other small or medium sized company can afford to throw at it. Most places you’d be lucky to find even one engineer. One engineer working full time with only a specific tool is a luxury and one such engineer will be able to achieve a lot.