4 ms·

My company is investing heavily in interactive runbooks (their own impl) and while I kinda dislike the idea of runbooks in general this is a nice implementation

My company is investing heavily in interactive runbooks (their own impl) and while I kinda dislike the idea of runbooks in general this is a nice implementation. Interactive Runbooks exist in some sort of tech-debt limbo where you have enough time to write a nice little runbook but apparently not enough time to properly automate whatever the task is. Ideally automation is easy and all change flows through a SDLC and you get all the nice things that come from that (testing for one). Instead companies build these cybernetic business processes via these adhoc notebooks to manually poke and prod production. This kinda made sense when they were these low-effort throw away things for rare events or short-term triage but now they're kinda-sorta code but without tests or reusability (or maybe you even further down the crazy well and write libraries and tests for your notebooks at which point you're really investing in the cybernetic operational model, and like, why?).

number6 3y ago
Wouldn't Ansible be the better runbook then?
- vladsanchez 3y ago
  I thought the same but about leaving Ansible for Runme. Not sure it's a wise idea but I like Runme DX/UX.
vineyardmike 3y ago
> Interactive Runbooks exist in some sort of tech-debt limbo where you have enough time to write a nice little runbook but apparently not enough time to properly automate whatever the task is I think this is a really common place to fall into. Automating 100% of conditions is hard, but having a google-doc full of bash commands is kinda meh. One of my previous teams decided on a "Maintainer Lambda" which would help with some tasks we couldn't automate fully. The Lambda could either make an action (trigger a db backup, trigger a bounce, etc) or it could perform some query (find FQDN hostname of server where ${INPUT} metric is in alarm). It'd help with calling complex APIs (eg. AWS APIs) repeatably, and testable, and it'd handle any permissions and input validations. The Lambda had unit testing, and it could run automatically under certain limited conditions. Over time, we've added to it, and let it handle more tasks, as long as it'd log it's actions in a ticket (eg. drain server, then auto-bounce on mem-full alarm). We'd embed cURL scripts into the runback doc, to trigger the functions manually. But the biggest win was whenever we got an automated ticket, for things like alarming metrics, it'd auto-run and parse the ticket, spitting out useful commands into the comments that the on-call could run (eg. copy+paste'able SSH-to-host commands based on which server triggered alarm). It's basically a reverse-run book. Instead of looking up what to do, we let the ticket tell us what to do, then slowly automate those steps.
- bberenberg 3y ago
  Sorry for off-topic, but was "runback" a typo? Because it happens to be a brand ... related to the sport of curling. Amazing coincidence if yes.
  - vineyardmike 3y ago
    Sorry. Supposed to be run book.
plaguuuuuu 3y ago
I have a bunch of things I want to use them for. amongst them * tutorials and other show and tell type stuff. * troubleshooting: have a collaboratively-authored runbook that queries APIs, loads/aggregates data in some way and allows people to interactively examine response data (like a debug terminal sort of thing but with graphing etc) I think they're really exciting.
PaulRobinson 3y ago
Runbooks are about eliminating toil incrementally. If you or your team find there's some process that needs to be done regularly, you're right, it should be automated. But there are often barriers to doing that immediately, and you may not even agree on what all the steps are. Here's how you do it then: 1. Write the runbook. 2. Execute the runbook multiple times. 3. Modify and improve the runbook. 4. Automate parts of the runbook. 5. When the entire thing is automated, hook it up to a metric to run automatically. 6. Delete your manual runbook - keep this new piece of code documented with its history and steps of improvement and eventual automation. This method means you get consistency of a method almost immediately, don't need to fight product to spend a ton of time on it (you can spend 30 minutes here and there on a single step), the team has better conversations about ad-hoc process, and you eventually automate it. In my job we have to do code releases to customers. This means we take a repo, do some tidying up like removing internal comments, inline some dependencies, make sure the README is in good shape, then make sure several people are notified and can approve it. I wrote a scrappy runbook with like 10 bullet points because this was being done really inconsistently. A month later it had been improved a ton by engineers with detailed command lines and roll back steps. Now they're starting to automate parts of it. We don't have explicit time for this, but we got a consistent process straight off the bat, engineer buy-in to the process and improving it, and bit by bit we will get to a point where it becomes a one-liner without us feeling we had to carve out a chunk of time specially for it. What's not to like?
- TheProTip 3y ago
  I worked at Rundeck on the core engineering team through the PagerDuty acquisition. Left my job start of the year to bootstrap a reboot, StepWise, that embraces and leans into that iterative approach to automation. We are looking for early adopters to try it out and give feedback soon here. Would love to get your thoughts; https://stepwisehq.com https://stepwisehq.com .
  - vladsanchez 3y ago
    Joined your waitlist. Looks and feels very intuitive and promising. Even more refined and polished than Rundeck and your competitors. Good job and thanks for sharing.

number6 3y ago

Wouldn't Ansible be the better runbook then?

vladsanchez 3y ago
I thought the same but about leaving Ansible for Runme. Not sure it's a wise idea but I like Runme DX/UX.

vineyardmike 3y ago

> Interactive Runbooks exist in some sort of tech-debt limbo where you have enough time to write a nice little runbook but apparently not enough time to properly automate whatever the task is I think this is a really common place to fall into. Automating 100% of conditions is hard, but having a google-doc full of bash commands is kinda meh. One of my previous teams decided on a "Maintainer Lambda" which would help with some tasks we couldn't automate fully. The Lambda could either make an action (trigger a db backup, trigger a bounce, etc) or it could perform some query (find FQDN hostname of server where ${INPUT} metric is in alarm). It'd help with calling complex APIs (eg. AWS APIs) repeatably, and testable, and it'd handle any permissions and input validations. The Lambda had unit testing, and it could run automatically under certain limited conditions. Over time, we've added to it, and let it handle more tasks, as long as it'd log it's actions in a ticket (eg. drain server, then auto-bounce on mem-full alarm). We'd embed cURL scripts into the runback doc, to trigger the functions manually. But the biggest win was whenever we got an automated ticket, for things like alarming metrics, it'd auto-run and parse the ticket, spitting out useful commands into the comments that the on-call could run (eg. copy+paste'able SSH-to-host commands based on which server triggered alarm). It's basically a reverse-run book. Instead of looking up what to do, we let the ticket tell us what to do, then slowly automate those steps.

bberenberg 3y ago
Sorry for off-topic, but was "runback" a typo? Because it happens to be a brand ... related to the sport of curling. Amazing coincidence if yes.
- vineyardmike 3y ago
  Sorry. Supposed to be run book.

plaguuuuuu 3y ago

I have a bunch of things I want to use them for. amongst them * tutorials and other show and tell type stuff. * troubleshooting: have a collaboratively-authored runbook that queries APIs, loads/aggregates data in some way and allows people to interactively examine response data (like a debug terminal sort of thing but with graphing etc) I think they're really exciting.

PaulRobinson 3y ago

Runbooks are about eliminating toil incrementally. If you or your team find there's some process that needs to be done regularly, you're right, it should be automated. But there are often barriers to doing that immediately, and you may not even agree on what all the steps are. Here's how you do it then: 1. Write the runbook. 2. Execute the runbook multiple times. 3. Modify and improve the runbook. 4. Automate parts of the runbook. 5. When the entire thing is automated, hook it up to a metric to run automatically. 6. Delete your manual runbook - keep this new piece of code documented with its history and steps of improvement and eventual automation. This method means you get consistency of a method almost immediately, don't need to fight product to spend a ton of time on it (you can spend 30 minutes here and there on a single step), the team has better conversations about ad-hoc process, and you eventually automate it. In my job we have to do code releases to customers. This means we take a repo, do some tidying up like removing internal comments, inline some dependencies, make sure the README is in good shape, then make sure several people are notified and can approve it. I wrote a scrappy runbook with like 10 bullet points because this was being done really inconsistently. A month later it had been improved a ton by engineers with detailed command lines and roll back steps. Now they're starting to automate parts of it. We don't have explicit time for this, but we got a consistent process straight off the bat, engineer buy-in to the process and improving it, and bit by bit we will get to a point where it becomes a one-liner without us feeling we had to carve out a chunk of time specially for it. What's not to like?

TheProTip 3y ago
I worked at Rundeck on the core engineering team through the PagerDuty acquisition. Left my job start of the year to bootstrap a reboot, StepWise, that embraces and leans into that iterative approach to automation. We are looking for early adopters to try it out and give feedback soon here. Would love to get your thoughts; https://stepwisehq.com https://stepwisehq.com .
- vladsanchez 3y ago
  Joined your waitlist. Looks and feels very intuitive and promising. Even more refined and polished than Rundeck and your competitors. Good job and thanks for sharing.