3 ms·
About 15 years ago, as part of our incident management process I wrote a wrapper around tmux + script that let SREs and other engineers share terminal sessions
by raffraffraff 3y ago
About 15 years ago, as part of our incident management process I wrote a wrapper around tmux + script that let SREs and other engineers share terminal sessions during incidents or complex / long-running operations. A feature I built in was the ability to easily record the whole session so we could capture "actual" actions (and not just the ones we could remember). Sure, you can type "history", but in a postmortem it's extremely useful to be able to play back 4 different terminals simultaneously and watch what each person did in real-time. We had a number of "sev2" incidents that we turned into "sev1" incidents while following runbooks.
To be honest, the main use of the script was helping engineers use tmux even if they had never had to before (y'know, mouse clicky, familiar window management keyboard shortcuts etc, automating the process of requesting access to a tmux session, receiving those notifications, and granting access). But I definitely improved a number of runbooks or found the "aha, that's how we broke it!" by watching those replays, in particular the timing of events in different terminal sessions.
An extremely useful feature in scriptreplay is the "remove all the long pauses" option.
- stavros 3y agoFYI, Tailscale does this natively now.