3 ms·

There's stuff where you can either spend weeks or months automating it - or live with the fact that your oncall engineer has to use a runbook a few times per ye

by wink 10y ago

There's stuff where you can either spend weeks or months automating it - or live with the fact that your oncall engineer has to use a runbook a few times per year to fix the problem. Also your Netflix example - how many people do you have there? Probably the smallest of teams is bigger than my company's whole engineering department. We're running a whole company and not a few services. (I'm absolutely not trying to discourage what you're doing - but I strongly feel it's a different ballgame.) The smaller your team with too many widespread responsibilities (not: make THIS service be available 99.9%, but make ALL services available 90%, then 95%, ...) the more you only automate what happens frequently. And yes, we try to first a basic runbook when something happens - and only if the same thing happens repeatedly, we automate it.

jedberg 10y ago
We were doing this with a team of sometimes one or two reliability engineers, but we were cheating, because our company culture meant that the engineers who built the systems are responsible for keeping them running, so they would invest their engineering time in fixing the problems along with us. I personally found that runbooks were even worse for small size teams (like our four person reddit team) because they would get out of data even quicker than at the bigger places due to the rapidly changing environment. I wrote down thread that if all of your deployment is automated than it is much easier to automate remediation, because you just change your deployment to fix the problem, as long as you can redeploy quickly.
- wink 10y ago
  I agree on one level (literally, not figuratively) - if you have awesome (or even bearable :P) deployment for your software stack. We can do that. On the other level, there's stuff we see as core infrastructure (for example Hardware, or even some parts of OpenStack running on that hardware) - of course there are also downtimes and emergencies and dumpster fires - but they are pretty much unique little snowflakes and the repeating ones happen a few times per year. There simply is nothing "to deploy". Maybe one can argue that "runbook" is not 100% correct, sometimes it's a runbook including debug info. But it's not turtles all the way down, and I stand to my point there's stuff where the cost to benefit ratio totally ends up at "automate it away" and there's other stuff where it's the opposite.
- bbrazil 10y ago
  > I personally found that runbooks were even worse for small size teams (like our four person reddit team) because they would get out of data even quicker than at the bigger places due to the rapidly changing environment. My experience is that you can get regularly updated runbooks, but they're at the wrong level of abstraction. They'll discuss some odd one-off failure that happened to trigger a given alert, rather than the general class of problems that this alert is trying to catch. These days I consider looking at a runbook to be an act of desperation, that'll I'll only perform after attempting to debug from first principles.
- toomuchtodo 10y ago
  > We were doing this with a team of sometimes one or two reliability engineers, but we were cheating, because our company culture meant that the engineers who built the systems are responsible for keeping them running, so they would invest their engineering time in fixing the problems along with us. What advice would you give for an org where the engineers who build systems are not responsible for keeping them running, and everyone on a (much smaller comparatively) infrastructure team is (which is slowly turning into an SRE team by necessity)? Anecdotally, I've found documentation to be useless; despite documentation being of a high quality, no one refers to it, even after iterating to continue to add information, make it more relevant, streamline, etc.
  - chimeracoder 10y ago
    > What advice would you give for an org where the engineers who build systems are not responsible for keeping them running I don't think that's a healthy or sustainable culture for a company, certainly not one that's expecting to grow. It might be sufficient for a company that has a small technical team and isn't looking to grow (think: the "tech department" for a company in another industry), but not for a company where engineering or technology is the primary focus.
    - toomuchtodo 10y ago
      Thanks for the reply!
  - jedberg 10y ago
    My advice would be to push as hard as you can to change the culture, or you'll be drowned. Engineers will not make it a priority to fix anything that causes outages because they will be evaluated on feature velocity, not uptime. If you can make the company culture focus on uptime, or get engineers involved in remediation, then you'll be better off. If you can't do that, try to at least push for the Google model: The engineers are responsible for uptime of their product until they can prove that it is stable and has sufficient monitoring and alerting, and then they can turn it over to SRE, with the caveat that it will go back to the engineers if it gets lower in quality.
problems 10y ago
Depends on the exact case, but I've managed to convert many-step deployments into single-step ansible playbooks in not much longer than it takes to run them once. Even at our super tiny 3-person operation this has saved a massive amount of time - it's even more valuable when it's panic time because a server went down due to disk or ISP failure and you know you have an ansible script that can get a new server up in 3 minutes flat while you grab the backup.

by wink 10y ago

jedberg 10y ago

We were doing this with a team of sometimes one or two reliability engineers, but we were cheating, because our company culture meant that the engineers who built the systems are responsible for keeping them running, so they would invest their engineering time in fixing the problems along with us. I personally found that runbooks were even worse for small size teams (like our four person reddit team) because they would get out of data even quicker than at the bigger places due to the rapidly changing environment. I wrote down thread that if all of your deployment is automated than it is much easier to automate remediation, because you just change your deployment to fix the problem, as long as you can redeploy quickly.

wink 10y ago
I agree on one level (literally, not figuratively) - if you have awesome (or even bearable :P) deployment for your software stack. We can do that. On the other level, there's stuff we see as core infrastructure (for example Hardware, or even some parts of OpenStack running on that hardware) - of course there are also downtimes and emergencies and dumpster fires - but they are pretty much unique little snowflakes and the repeating ones happen a few times per year. There simply is nothing "to deploy". Maybe one can argue that "runbook" is not 100% correct, sometimes it's a runbook including debug info. But it's not turtles all the way down, and I stand to my point there's stuff where the cost to benefit ratio totally ends up at "automate it away" and there's other stuff where it's the opposite.
bbrazil 10y ago
> I personally found that runbooks were even worse for small size teams (like our four person reddit team) because they would get out of data even quicker than at the bigger places due to the rapidly changing environment. My experience is that you can get regularly updated runbooks, but they're at the wrong level of abstraction. They'll discuss some odd one-off failure that happened to trigger a given alert, rather than the general class of problems that this alert is trying to catch. These days I consider looking at a runbook to be an act of desperation, that'll I'll only perform after attempting to debug from first principles.
toomuchtodo 10y ago
> We were doing this with a team of sometimes one or two reliability engineers, but we were cheating, because our company culture meant that the engineers who built the systems are responsible for keeping them running, so they would invest their engineering time in fixing the problems along with us. What advice would you give for an org where the engineers who build systems are not responsible for keeping them running, and everyone on a (much smaller comparatively) infrastructure team is (which is slowly turning into an SRE team by necessity)? Anecdotally, I've found documentation to be useless; despite documentation being of a high quality, no one refers to it, even after iterating to continue to add information, make it more relevant, streamline, etc.
- chimeracoder 10y ago
  > What advice would you give for an org where the engineers who build systems are not responsible for keeping them running I don't think that's a healthy or sustainable culture for a company, certainly not one that's expecting to grow. It might be sufficient for a company that has a small technical team and isn't looking to grow (think: the "tech department" for a company in another industry), but not for a company where engineering or technology is the primary focus.
  - toomuchtodo 10y ago
    Thanks for the reply!
- jedberg 10y ago
  My advice would be to push as hard as you can to change the culture, or you'll be drowned. Engineers will not make it a priority to fix anything that causes outages because they will be evaluated on feature velocity, not uptime. If you can make the company culture focus on uptime, or get engineers involved in remediation, then you'll be better off. If you can't do that, try to at least push for the Google model: The engineers are responsible for uptime of their product until they can prove that it is stable and has sufficient monitoring and alerting, and then they can turn it over to SRE, with the caveat that it will go back to the engineers if it gets lower in quality.

problems 10y ago

Depends on the exact case, but I've managed to convert many-step deployments into single-step ansible playbooks in not much longer than it takes to run them once. Even at our super tiny 3-person operation this has saved a massive amount of time - it's even more valuable when it's panic time because a server went down due to disk or ISP failure and you know you have an ansible script that can get a new server up in 3 minutes flat while you grab the backup.