3 ms·
I highly recommend against runbooks. There should be no need for runbooks. Anything that can be identified down to a single solution should have an automated re
by GauntletWizard 3y ago
I highly recommend against runbooks. There should be no need for runbooks. Anything that can be identified down to a single solution should have an automated response.
What you should have is hintbooks. For each alert, have something that identifies why you made the alert. What dashboards to check, what logs to read, what else might be going wrong.
- ryanklee 3y agoAutomated responses take time and tech to implement. Playbooks are a way to iterate into automation. Good automated solutions often don't emerge fully baked.
- GauntletWizard 3y agoBy the time you have enough clarity to create a "playbook", you probably have enough clarity to reproduce and to automate. That said, there are still a ton of solutions where "playbook" is kind of the right call. "DB is out of Disk Space" is probably "Increase the size of the disk reservation" for most modern cloud-native companies. They rarely rise to the level of detail that proponents of "Playbooks" suggest, though. How do you do that? Is it clicking through the UI? Do you have the CLI command prepared? Have you thought about why disk space usage has grown suddenly? This is the sort of thing that may be a no brainer you were expecting (in which case - Why didn't you complete the operation in-hours?) or it could be the "That's funny[1]" moment that precedes a major discovery. [1] https://quoteinvestigator.com/2015/03/02/eureka-funny/ https://quoteinvestigator.com/2015/03/02/eureka-funny/