> Interactive Runbooks exist in some sort of tech-debt limbo where you have enough time to write a nice little runbook but apparently not enough time to properl

> Interactive Runbooks exist in some sort of tech-debt limbo where you have enough time to write a nice little runbook but apparently not enough time to properly automate whatever the task is I think this is a really common place to fall into. Automating 100% of conditions is hard, but having a google-doc full of bash commands is kinda meh. One of my previous teams decided on a "Maintainer Lambda" which would help with some tasks we couldn't automate fully. The Lambda could either make an action (trigger a db backup, trigger a bounce, etc) or it could perform some query (find FQDN hostname of server where ${INPUT} metric is in alarm). It'd help with calling complex APIs (eg. AWS APIs) repeatably, and testable, and it'd handle any permissions and input validations. The Lambda had unit testing, and it could run automatically under certain limited conditions. Over time, we've added to it, and let it handle more tasks, as long as it'd log it's actions in a ticket (eg. drain server, then auto-bounce on mem-full alarm). We'd embed cURL scripts into the runback doc, to trigger the functions manually. But the biggest win was whenever we got an automated ticket, for things like alarming metrics, it'd auto-run and parse the ticket, spitting out useful commands into the comments that the on-call could run (eg. copy+paste'able SSH-to-host commands based on which server triggered alarm). It's basically a reverse-run book. Instead of looking up what to do, we let the ticket tell us what to do, then slowly automate those steps.