4 ms·

> Runbooks are more of an anti-pattern than anything. No. They're essential to the sanity of the individual SRE oncall, and of the SRE team they are part of. I

> Runbooks are more of an anti-pattern than anything. No. They're essential to the sanity of the individual SRE oncall, and of the SRE team they are part of. It's also how institutional knowledge is preserved after successive rounds of oncall shifts. If you think a playbook is bad, try being oncall for the first time for a massively (Google-scale) distributed system without a playbook. > A list of commands you should run to accomplish X -> that's a script. A list of possible root causes an SRE should consider, complex interactions with other systems that might be problematic, not overreacting to spurious alerts -> that's a playbook.

notyourday 7y ago
> If you think a playbook is bad, try being oncall for the first time for a massively (Google-scale) distributed system without a playbook. You are not Google scale. Don't invent a pen that writes in space when a pencil would do the trick.
- marcinzm 7y ago
  I'd argue 100% automation is a space pen while run books are a pencil.
  - notyourday 7y ago
    You do not need 100% automation. What you need is a systematic approach to handling problems followed by fixing the root cause. Runbooks came from techops in broadcasting, power plant operations, etc where there was a clear division between operators who pushed buttons, ran cables, etc and those that made decisions about buttons to push and cables to run. Dumb hands + runbooks created "smart hands". If your SRE runs like that it is not SRE. Look at the incident handling: 1. Identify the issue 2. Implement a workaround to restore the service 3. Identify the root cause 4. Implement a fix for the root cause 5. Remove the workaround Runbooks cover 1. and 2.
- easymodex 7y ago
  Did you just assume my scale?
- p_l 7y ago
  And check your metaphors before using them. The pencil was more expensive.
- ggambetta 7y ago
  > You are not Google scale. I don't know, I think I was kind of Google scale when working as a Google SRE.
  - notyourday 7y ago
    1) https://www.transposit.com/ https://www.transposit.com/ is not Google. 2) Your impact on Google scale was nearly 0 because you were one of a thousand SREs at Google.
    - williamDafoe 7y ago
      There are only 500.
      - notyourday 7y ago
        My bad
    - GaryNumanVevo 7y ago
      There’s no tolerance for rude and unsubstantiated comments here
0x445442 7y ago
It seems the problems your describing are management problems not engineering problems. The people who designed, built and are responsible for maintaining and updating the system should be on call if something goes wrong. I view runbooks in the same way I view "knowledge transfers". It's management's desperate hope that somehow a person leaving a project or company can convey their acquired knowledge of a system on some Confluence page or from a few meetings. It's a complete failure to recognizing the essence of the work. I don't think runbooks are necessarily bad if they are used to bootstrap new team members. But thinking their existence is a green light to allow people without domain expertise in a system to be responsible for administering the system during "off hours" is misguided.
- ggambetta 7y ago
  > The people who designed, built and are responsible for maintaining and updating the system should be on call if something goes wrong. In my experience, the devs were somewhere in the US west coast, and the SRE teams were geographically distributed to cover the 24 hour period during local daytime (nobody likes to be paged in the middle of the night). As an SRE in Zürich, I got paged in what was the middle of the night for the Kirkland people, dealt with the emergency (using the playbook), root-caused it (with the assistance of the playbook), and filed bugs to be looked at by the dev team when they woke up. The systems stayed up, everyone could sleep at night, working as intended.
  - 0x445442 7y ago
    > and the SRE teams were geographically distributed to cover the 24 hour period during local daytime Management problem number 1. These people should not be responsible for the running system. > nobody likes to be paged in the middle of the night Excellent motivation for the people that should be responsible for the running system to build quality software.
    - SpicyLemonZest 7y ago
      Why would randomly waking your engineers up in the middle of the night be an excellent motivation strategy?

notyourday 7y ago

> If you think a playbook is bad, try being oncall for the first time for a massively (Google-scale) distributed system without a playbook. You are not Google scale. Don't invent a pen that writes in space when a pencil would do the trick.

marcinzm 7y ago
I'd argue 100% automation is a space pen while run books are a pencil.
- notyourday 7y ago
  You do not need 100% automation. What you need is a systematic approach to handling problems followed by fixing the root cause. Runbooks came from techops in broadcasting, power plant operations, etc where there was a clear division between operators who pushed buttons, ran cables, etc and those that made decisions about buttons to push and cables to run. Dumb hands + runbooks created "smart hands". If your SRE runs like that it is not SRE. Look at the incident handling: 1. Identify the issue 2. Implement a workaround to restore the service 3. Identify the root cause 4. Implement a fix for the root cause 5. Remove the workaround Runbooks cover 1. and 2.
easymodex 7y ago
Did you just assume my scale?
p_l 7y ago
And check your metaphors before using them. The pencil was more expensive.
ggambetta 7y ago
> You are not Google scale. I don't know, I think I was kind of Google scale when working as a Google SRE.
- notyourday 7y ago
  1) https://www.transposit.com/ https://www.transposit.com/ is not Google. 2) Your impact on Google scale was nearly 0 because you were one of a thousand SREs at Google.
  - williamDafoe 7y ago
    There are only 500.
    - notyourday 7y ago
      My bad
  - GaryNumanVevo 7y ago
    There’s no tolerance for rude and unsubstantiated comments here

0x445442 7y ago

It seems the problems your describing are management problems not engineering problems. The people who designed, built and are responsible for maintaining and updating the system should be on call if something goes wrong. I view runbooks in the same way I view "knowledge transfers". It's management's desperate hope that somehow a person leaving a project or company can convey their acquired knowledge of a system on some Confluence page or from a few meetings. It's a complete failure to recognizing the essence of the work. I don't think runbooks are necessarily bad if they are used to bootstrap new team members. But thinking their existence is a green light to allow people without domain expertise in a system to be responsible for administering the system during "off hours" is misguided.

ggambetta 7y ago
> The people who designed, built and are responsible for maintaining and updating the system should be on call if something goes wrong. In my experience, the devs were somewhere in the US west coast, and the SRE teams were geographically distributed to cover the 24 hour period during local daytime (nobody likes to be paged in the middle of the night). As an SRE in Zürich, I got paged in what was the middle of the night for the Kirkland people, dealt with the emergency (using the playbook), root-caused it (with the assistance of the playbook), and filed bugs to be looked at by the dev team when they woke up. The systems stayed up, everyone could sleep at night, working as intended.
- 0x445442 7y ago
  > and the SRE teams were geographically distributed to cover the 24 hour period during local daytime Management problem number 1. These people should not be responsible for the running system. > nobody likes to be paged in the middle of the night Excellent motivation for the people that should be responsible for the running system to build quality software.
  - SpicyLemonZest 7y ago
    Why would randomly waking your engineers up in the middle of the night be an excellent motivation strategy?