7 ms·

Incident management at Google – adventures in SRE-land

250 points by kungfudoi 10y ago

mikecb 10y ago
The coolest thing I took away from the SRE book was this progression of system operations from manual, to scriptable, to automated, to a fourth category I hadn't even known existed: autonomous. The idea that you can keep moving up this hierarchy of exception management beyond even chef and puppet, and systems will be able to heal themselves, is a pretty cool one. As a manager, this made the concept of 20% time a lot more clear. These are people with the knowledge and incentive to build a hierarchy of systems that progressively remove risk from their work. This is in fact their primary business objective. And we need to make sure they have time to do that, vs working them to death with manual remediation. It's a great lesson. Incidentally, Stackdriver contains a simple alerting and incident management tool that's really nice to use. Hopefully it gets more robust as time goes on and larger and more complex orgs move to their cloud. Edit: not Outalator.
- bluedino 10y ago
  >> from manual, to scriptable, to automated, to a fourth category I hadn't even known existed: autonomous. What's the difference between automated and autonomous?
  - mikecb 10y ago
    I can't say it better than they did: https://landing.google.com/sre/book/chapters/automation-at-google.html https://landing.google.com/sre/book/chapters/automation-at-g...
  - xapata 10y ago
    Autonomous does not require any human interaction, not even to trigger a script. In contrast, "automated" systems might still require a human to start the process or have a very simplistic trigger, such as based on the time of day.
- ben_jones 10y ago
  If anything it proves how much mismanagement / wasted potential software organizations have had in the last 20 years. Full automation should be the natural progression of our trade but I fear most companies stall after 5ish years due to turnover, brain-drain, re-organizations, acquisitions, management incompetency etc. Google on the other hand has always had a seemingly never ending pool of resources and talent to keep pushing the barrier further. Fortunately they give a lot back to the community in the form of books, talks, and projects such as Kubernetes (a poor man's Borg). However I fear that with all things commercial it will lead to an oligarchy where companies like Google, Facebook, and Uber, are just that far ahead of the curve nobody else will ever catch up.
- kyrra 10y ago
  I'm a google employee, opinions are my own. The incident management tool is not Outalator. Outalator is the pager queue management tool. The incident management tool is for manually creating incidents that have much broader visibility than Outlator does. As someone who has been incident commander a few times, incidents tend to have broader impact beyond your immediate team or owned jobs.
  - mikecb 10y ago
    Thanks, my bad. Would you say the stackdriver incidents tool was influenced by SRE experiences?
    - advisedwang 10y ago
      The Stackdriver incidents tool existed before it Stackdriver was bought by Google. They do have similarities though.
- nodesocket 10y ago
  progression of system operations from manual, to scriptable, to automated, to a fourth category I hadn't even known existed: autonomous. Completely agree. The first "eureka" moment is when you define all your infrastructure in code in something like Terraform. Magically networking, firewall rules, disks, instances, are all provisioned and dependencies calculated. It is quite a breakthrough from running CLI commands or using the web interface to allocate infrastructure. Plug: I wrote a blog post on getting started with Terraform and Google Compute Engine for those interetested https://blog.elasticbyte.net/getting-started-with-terraform-and-google-compute-engine/ https://blog.elasticbyte.net/getting-started-with-terraform-...
- asuffield 10y ago
  (I'm a Google SRE. My opinions are my own.) That's not what our 20% time is for, and 20% is way too small a number for that purpose. "20% time" (the way we use the term) is for personal/career growth/scratching itches. Time spent on building systems that make our service better is my primary job. Manual remediation ("toil") is something to be tracked as a dangerous antipattern that must not be allowed to take over. Toil and oncall response should be less than 20% of my time, together. At least half my time should go into engineering projects. If the level of toil is in excess of 50% of team activity then I would expect only percussive intervention to get the team out of this situation.
  - mikecb 10y ago
    Great comment, thanks for the clarification. Wasn't trying to say that 20% is a magic number, just that it cemented the idea for me that engineering time, and self-directed engineering time, is incredibly valuable for everyone that can be justified and should be zealously protected.
  - deepakhj 10y ago
    How do you track how much time you spend on manual vs projects? Is there an in house tracking tool?
no_wizard 10y ago
I find this bit to be particularly insightful: "Can I handle this? What if I can’t?" But then I started to work the problem in front of me, like I was trained to, and I remembered that I don’t need to know everything — there are other people I can call on, and they will answer. I may be on point, but I’m not alone It might be because I'm currently training people in this realm, and this is one of their biggest fears, or maybe because it was my biggest fear, but its so true. We're a team. We're here to help. At least if your SRE org is any good. Never be afraid to ask for help, and never be afraid to admit you don't know something or it might be outside of your comfort zone. I'll take willing to learn and readily able to admit knowledge deficits over someone who doesn't any day of the week. Great book they're working on, great article on this. So many gems, but this one stuck out for me, and its pretty relevant to me right now.
- HerraBRE 10y ago
  It's not mentioned in the article, but there is an underlying point that affects hiring for roles like this: you need people who can and will admit they don't know everything and will ask for help rather than wing it. "Rock stars" are downright dangerous, as are people who prefer to make things up rather than admit ignorance. A new SRE doesn't need to know everything (and can't). But he absolutely needs to be curious and willing to ask for help.
  - btmiller 10y ago
    It's an interesting dilemma in my mind. I remember reading through the SRE book about how SREs a required to have both depth and breadth. Seems like a nearly impossible target to hit IMO, so how are you supposed to simultaneously reconcile deep/broad abilities with humility when hiring for SREs?
    - user5994461 10y ago
      You don't, you just wait and struggle to find the 0.00x % of people who fits the role. There is a reason that this is an impossible to fill role.
    - 10y ago
twosheep 10y ago
Maybe it's just me but I found the constant in-line plugs for the book to be distracting -- footnotes would have been better. Interesting write-up though
- daenney 10y ago
  I had a similar reaction. I was a bit irked by it b/c it felt very pushy towards the SRE book and broke me out of the flow of the article a few times. 10/10 on the book though, would recommend anyone to read it.
WestCoastJustin 10y ago
FYI - it's linked to in the post, but in case it's not obvious, they have posted the SRE book for free at https://landing.google.com/sre/book.html https://landing.google.com/sre/book.html I'd highly recommend it if you're in the Ops feild. Probably the best book out there on current large scale Ops practices.
- ben_jones 10y ago
  It's a great book, very well written and fair. But the first time I read it I suffered a certain amount of zealotry: "Google is amazing! I should rewrite everything to be more like them!". Really a subcaption everyone should keep in mind is that the book defines how Google built systems for Google. YMMV.
  - Tushon 10y ago
    Currently reading the book, and could see the zealotry come out, but they mention, repeatedly, that your own systems may not need the level of service that Google built-in, or your teams may not be big enough to justify, etc. I don't think it is quite fair to indicate that they didn't give that thought. I totally agree that it is a great book and should be encouraged throughout ops orgs, devops roles, and developers who want to plan for the future, but that not every company needs the Google way to 100%, or maybe even 50% (as you indicated as well).
  - tbrowbdidnso 10y ago
    Also keep in mind that even the mighty Google has some rusty and pointy internal tools. There's no reason to mention any crappy things in a book about good practices ;).
saycheese 10y ago
Anyone able to compare and contrast Google's "Wheel of Misfortune" with Netflix's "Chaos Monkey" both in terms of the systems that enable them and the operations that relate to them?
- thesandlord 10y ago
  Chaos Monkey is more like Google's DiRT http://queue.acm.org/detail.cfm?id=2371516 http://queue.acm.org/detail.cfm?id=2371516
  - Kostchei 10y ago
    and Dust
    - grf 10y ago
      Not a thing anymore for several years, i.e name consolidation. Also, a good chunk of DiRT is now continuous and automated (not autonomous though). Disclaimer: I work at Google and ran the DiRT team for a few years incl. incident management itself.
- bskap 10y ago
  They're unrelated. Wheel of Misfortune is just a role-playing replay of a previous incident as a training exercise. Someone will grab (or simulate) logs and dashboards from the incident and then play GM for the wheel of misfortune at a future team meeting. Someone who isn't familiar with the incident will be designated "on-call". They'll state what they want to do and the GM will tell them or show them what they see when they do those things. Chaos Monkey is actually taking down production systems to make sure the system as a whole stays up when those individual pieces fall. Google does have (manual, not automatic) exercises doing similar things called DiRT (Disaster Recovery Testing), but it's not related to the SRE training exercise. (standard disclaimer: Google employee, not speaking for company, all opinions are my own, etc.)
- asuffield 10y ago
  (I'm an SRE at Google. My opinions are my own.) WoMs are a training exercise, intended to build familiarity with systems and how to respond when oncall. A typical WoM format is a few SREs sat in a room, with a designated victim who is pretending to be oncall. The person running the WoM will open with an exchange a bit like this (massively simpified): "You receive a page with this alert in it showing suddenly elevated rpc errors (link/paste)" "I'm going to look at this console to see if there was just a rollout" "Okay, you see a rollout happened about two minutes before the spike in rpc errors" "I'll roll that back in one location" "rpc errors go back to normal in that location" ...etc (Depending on the team and quality of simulation available, some of this may be replaced with actual historical monitoring data or simulated broken systems) The "chaos monkey" tool, as I understand it, is intended to maintain a minimum level of failure in order to make sure that failure cases are exercised. I've never been on a team which needed one of those: at sufficient scale and development velocity, the baseline rate of naturally occurring failures is already high enough. We do have some tools like that, but they're more commonly used by the dev teams during testing (where the test environment won't be big enough to naturally experience all the failures that happen in production).
vgy7ujm 10y ago
Is it just me or are we seeing a trend almost before the "new" role SRE has become mainstream that SRE is turning into support technicians because that is what is needed at most places that are not Google scale. The devaluation of the sysadmin took some time, this is happening much faster. What will be the next title when SRE can't get you a decent salary anymore? And why don't we see the same with the SWE role? Is it just that business leaders sees Ops as cost no matter what name it has?

250 points by kungfudoi 10y ago

The coolest thing I took away from the SRE book was this progression of system operations from manual, to scriptable, to automated, to a fourth category I hadn't even known existed: autonomous. The idea that you can keep moving up this hierarchy of exception management beyond even chef and puppet, and systems will be able to heal themselves, is a pretty cool one. As a manager, this made the concept of 20% time a lot more clear. These are people with the knowledge and incentive to build a hierarchy of systems that progressively remove risk from their work. This is in fact their primary business objective. And we need to make sure they have time to do that, vs working them to death with manual remediation. It's a great lesson. Incidentally, Stackdriver contains a simple alerting and incident management tool that's really nice to use. Hopefully it gets more robust as time goes on and larger and more complex orgs move to their cloud. Edit: not Outalator.

bluedino 10y ago
>> from manual, to scriptable, to automated, to a fourth category I hadn't even known existed: autonomous. What's the difference between automated and autonomous?
- mikecb 10y ago
  I can't say it better than they did: https://landing.google.com/sre/book/chapters/automation-at-google.html https://landing.google.com/sre/book/chapters/automation-at-g...
- xapata 10y ago
  Autonomous does not require any human interaction, not even to trigger a script. In contrast, "automated" systems might still require a human to start the process or have a very simplistic trigger, such as based on the time of day.
ben_jones 10y ago
If anything it proves how much mismanagement / wasted potential software organizations have had in the last 20 years. Full automation should be the natural progression of our trade but I fear most companies stall after 5ish years due to turnover, brain-drain, re-organizations, acquisitions, management incompetency etc. Google on the other hand has always had a seemingly never ending pool of resources and talent to keep pushing the barrier further. Fortunately they give a lot back to the community in the form of books, talks, and projects such as Kubernetes (a poor man's Borg). However I fear that with all things commercial it will lead to an oligarchy where companies like Google, Facebook, and Uber, are just that far ahead of the curve nobody else will ever catch up.
kyrra 10y ago
I'm a google employee, opinions are my own. The incident management tool is not Outalator. Outalator is the pager queue management tool. The incident management tool is for manually creating incidents that have much broader visibility than Outlator does. As someone who has been incident commander a few times, incidents tend to have broader impact beyond your immediate team or owned jobs.
- mikecb 10y ago
  Thanks, my bad. Would you say the stackdriver incidents tool was influenced by SRE experiences?
  - advisedwang 10y ago
    The Stackdriver incidents tool existed before it Stackdriver was bought by Google. They do have similarities though.
nodesocket 10y ago
progression of system operations from manual, to scriptable, to automated, to a fourth category I hadn't even known existed: autonomous. Completely agree. The first "eureka" moment is when you define all your infrastructure in code in something like Terraform. Magically networking, firewall rules, disks, instances, are all provisioned and dependencies calculated. It is quite a breakthrough from running CLI commands or using the web interface to allocate infrastructure. Plug: I wrote a blog post on getting started with Terraform and Google Compute Engine for those interetested https://blog.elasticbyte.net/getting-started-with-terraform-and-google-compute-engine/ https://blog.elasticbyte.net/getting-started-with-terraform-...
asuffield 10y ago
(I'm a Google SRE. My opinions are my own.) That's not what our 20% time is for, and 20% is way too small a number for that purpose. "20% time" (the way we use the term) is for personal/career growth/scratching itches. Time spent on building systems that make our service better is my primary job. Manual remediation ("toil") is something to be tracked as a dangerous antipattern that must not be allowed to take over. Toil and oncall response should be less than 20% of my time, together. At least half my time should go into engineering projects. If the level of toil is in excess of 50% of team activity then I would expect only percussive intervention to get the team out of this situation.
- mikecb 10y ago
  Great comment, thanks for the clarification. Wasn't trying to say that 20% is a magic number, just that it cemented the idea for me that engineering time, and self-directed engineering time, is incredibly valuable for everyone that can be justified and should be zealously protected.
- deepakhj 10y ago
  How do you track how much time you spend on manual vs projects? Is there an in house tracking tool?

no_wizard 10y ago

I find this bit to be particularly insightful: "Can I handle this? What if I can’t?" But then I started to work the problem in front of me, like I was trained to, and I remembered that I don’t need to know everything — there are other people I can call on, and they will answer. I may be on point, but I’m not alone It might be because I'm currently training people in this realm, and this is one of their biggest fears, or maybe because it was my biggest fear, but its so true. We're a team. We're here to help. At least if your SRE org is any good. Never be afraid to ask for help, and never be afraid to admit you don't know something or it might be outside of your comfort zone. I'll take willing to learn and readily able to admit knowledge deficits over someone who doesn't any day of the week. Great book they're working on, great article on this. So many gems, but this one stuck out for me, and its pretty relevant to me right now.

HerraBRE 10y ago
It's not mentioned in the article, but there is an underlying point that affects hiring for roles like this: you need people who can and will admit they don't know everything and will ask for help rather than wing it. "Rock stars" are downright dangerous, as are people who prefer to make things up rather than admit ignorance. A new SRE doesn't need to know everything (and can't). But he absolutely needs to be curious and willing to ask for help.
- btmiller 10y ago
  It's an interesting dilemma in my mind. I remember reading through the SRE book about how SREs a required to have both depth and breadth. Seems like a nearly impossible target to hit IMO, so how are you supposed to simultaneously reconcile deep/broad abilities with humility when hiring for SREs?
  - user5994461 10y ago
    You don't, you just wait and struggle to find the 0.00x % of people who fits the role. There is a reason that this is an impossible to fill role.
  - 10y ago

twosheep 10y ago

Maybe it's just me but I found the constant in-line plugs for the book to be distracting -- footnotes would have been better. Interesting write-up though

daenney 10y ago
I had a similar reaction. I was a bit irked by it b/c it felt very pushy towards the SRE book and broke me out of the flow of the article a few times. 10/10 on the book though, would recommend anyone to read it.

WestCoastJustin 10y ago

FYI - it's linked to in the post, but in case it's not obvious, they have posted the SRE book for free at https://landing.google.com/sre/book.html https://landing.google.com/sre/book.html I'd highly recommend it if you're in the Ops feild. Probably the best book out there on current large scale Ops practices.

ben_jones 10y ago
It's a great book, very well written and fair. But the first time I read it I suffered a certain amount of zealotry: "Google is amazing! I should rewrite everything to be more like them!". Really a subcaption everyone should keep in mind is that the book defines how Google built systems for Google. YMMV.
- Tushon 10y ago
  Currently reading the book, and could see the zealotry come out, but they mention, repeatedly, that your own systems may not need the level of service that Google built-in, or your teams may not be big enough to justify, etc. I don't think it is quite fair to indicate that they didn't give that thought. I totally agree that it is a great book and should be encouraged throughout ops orgs, devops roles, and developers who want to plan for the future, but that not every company needs the Google way to 100%, or maybe even 50% (as you indicated as well).
- tbrowbdidnso 10y ago
  Also keep in mind that even the mighty Google has some rusty and pointy internal tools. There's no reason to mention any crappy things in a book about good practices ;).

saycheese 10y ago

Anyone able to compare and contrast Google's "Wheel of Misfortune" with Netflix's "Chaos Monkey" both in terms of the systems that enable them and the operations that relate to them?

thesandlord 10y ago
Chaos Monkey is more like Google's DiRT http://queue.acm.org/detail.cfm?id=2371516 http://queue.acm.org/detail.cfm?id=2371516
- Kostchei 10y ago
  and Dust
  - grf 10y ago
    Not a thing anymore for several years, i.e name consolidation. Also, a good chunk of DiRT is now continuous and automated (not autonomous though). Disclaimer: I work at Google and ran the DiRT team for a few years incl. incident management itself.
bskap 10y ago
They're unrelated. Wheel of Misfortune is just a role-playing replay of a previous incident as a training exercise. Someone will grab (or simulate) logs and dashboards from the incident and then play GM for the wheel of misfortune at a future team meeting. Someone who isn't familiar with the incident will be designated "on-call". They'll state what they want to do and the GM will tell them or show them what they see when they do those things. Chaos Monkey is actually taking down production systems to make sure the system as a whole stays up when those individual pieces fall. Google does have (manual, not automatic) exercises doing similar things called DiRT (Disaster Recovery Testing), but it's not related to the SRE training exercise. (standard disclaimer: Google employee, not speaking for company, all opinions are my own, etc.)
asuffield 10y ago
(I'm an SRE at Google. My opinions are my own.) WoMs are a training exercise, intended to build familiarity with systems and how to respond when oncall. A typical WoM format is a few SREs sat in a room, with a designated victim who is pretending to be oncall. The person running the WoM will open with an exchange a bit like this (massively simpified): "You receive a page with this alert in it showing suddenly elevated rpc errors (link/paste)" "I'm going to look at this console to see if there was just a rollout" "Okay, you see a rollout happened about two minutes before the spike in rpc errors" "I'll roll that back in one location" "rpc errors go back to normal in that location" ...etc (Depending on the team and quality of simulation available, some of this may be replaced with actual historical monitoring data or simulated broken systems) The "chaos monkey" tool, as I understand it, is intended to maintain a minimum level of failure in order to make sure that failure cases are exercised. I've never been on a team which needed one of those: at sufficient scale and development velocity, the baseline rate of naturally occurring failures is already high enough. We do have some tools like that, but they're more commonly used by the dev teams during testing (where the test environment won't be big enough to naturally experience all the failures that happen in production).

vgy7ujm 10y ago

Is it just me or are we seeing a trend almost before the "new" role SRE has become mainstream that SRE is turning into support technicians because that is what is needed at most places that are not Google scale. The devaluation of the sysadmin took some time, this is happening much faster. What will be the next title when SRE can't get you a decent salary anymore? And why don't we see the same with the SWE role? Is it just that business leaders sees Ops as cost no matter what name it has?