4 ms·
Show HN: PlayBooks – Jupyter Notebooks style on-call investigation documents
Hello everyone, Dipesh and Siddarth here. We are building PlayBooks (https://github.com/DrDroidLab/playbooks https://github.com/DrDroidLab/playbooks), an open source tool to write executable notebooks for on-call investigations / remediations instead of Google Docs or Wikis. There’s a demo video here: https://www.youtube.com/watch?v=_e-wOtIm1gk https://www.youtube.com/watch?v=_e-wOtIm1gk, and our docs are here: https://docs.drdroid.io/docs/playbooks https://docs.drdroid.io/docs/playbooks
We were in YC’s W23 batch working on a data lakehouse with support for dynamic log schemas. Eventually we realized it was a product in search of a market and decided to stop building it. When pivoting, we decided to work on something that we originally prototyped (before even YC) but didn’t execute on.
In our previous jobs, we were at a food delivery startup in India with a busy on-call routine for backend & devops engineers and a small tech team. Often business impacting issues (e.g. orders dropped by >5% in the last 15 minutes) would escalate to Dipesh as he was the lead dev who had been around for a while and he always had 4-5 hypotheses on what might have failed. To avoid becoming the bottleneck, he used to write scripts that fetched custom metrics & order related application logs every 5 minutes during peak traffic. So if an issue was reported, engineers would check the output of those scripts with all the usual suspects first, before diving into a generic exploration. This was the inspiration to get started on PlayBooks.
We’ve put together a platform that can help any dev create scripts with flexibility and without requiring to code much. Our goals were: (1) it can be automated to run and send updates; (2) investigation progress can be shared easily with other team members so everyone has the right context; (3) It can all be done without being on-call or having a laptop access.
Using PlayBooks, a user can configure the steps as data queries or actions within their observability stack. Here are the integrations we currently support: - Run bash commands on a remote server; - Fetch logs from AWS Cloudwatch and Azure Log Analytics; - Fetch metrics from any PromQL compatible db, AWS Cloudwatch, Datadog and New Relic; - Query PostgreSQL, ClickHouse or any other JDBC compatible databases; - Write a custom API call; - Query events from EKS / GKE; - Add an iFrame
The platform focuses on not just running the tasks but also displaying information in a meaningful form with relevant graphs / logs / text outputs alongside the steps in a notebook format. Some of our users have shared feedback that on-call decision making overload has reduced with PlayBooks as relevant data from multiple tools is presented upfront in one page.
Here are some of the key features that we believe will further increase the value to users looking to improve developer experience for their on-call engineers: - Automated surfacing of PlayBooks against alerts & enriching alerts with above-mentioned data; - AI-supported interpretation layer — connect with LLM or ML models to auto-analyze the data in the playbook; - Logs of historical executions to ease the effort of creating post-mortems / timelines and/or share information with peers.
If this looks like something that would have been useful for you on-call or will be in your current workspace, we welcome you to try our sandbox: https://sandbox.drdroid.io/ https://sandbox.drdroid.io/. We have added a default playbook. Just click on one of the steps in the playbook and then the “Run” button to see the playbook in action.
We are excited to hear what you like about the PlayBooks and what you think could improve the oncall developer experience for your team. Please drop your comments here – we will read them eagerly and respond!
- vvoruganti 2y agoThis is really cool! Love seeing more tools to help SREs and hopefully lessen the burden of on calls. The notebook style interface for logging and taking notes is appealing too. Seen a similar approach with https://fiberplane.com/ https://fiberplane.com/ Haven't been able to play around too much but watching the space
- TheBengaluruGuy 2y agoThank you. If you get a chance to play around, would love to hear your thoughts on it :)
- taeric 2y agoReminds me of https://nathanielhoag.com/blog/2022/interactive-runbook/ https://nathanielhoag.com/blog/2022/interactive-runbook/. Fun space to play in. Good luck on this!
- TheBengaluruGuy 2y agoThis is quite an interesting evaluation, thanks for sharing. We are piloting with a large enterprise (100+ SREs). Before us, they started implementing Jupyter Notebooks in a similar direction. Writing one playbook in Jupyter is easy but creating a framework to enable their 100+ product teams to self-serve and create playbooks has been so intensive for them, they even started working on their internal SDK for it. It was a lot of code and the lead felt like the Jupyter visual interface was harder to follow for instructions/runbooks. With PlayBooks, we have tried to abstract out the entire execution engine and configuration to a intuitive user experience (our architecture is explained here -- https://slender-resolution-789.notion.site/PlayBooks-Documentation-049a30663f3040b4ab85659552476e81 https://slender-resolution-789.notion.site/PlayBooks-Documen... )
- anonme4ever 2y agoYou should check out Nurtch[0] with Rubix integration[1]. Gitlab have some docs on how to use it[2]. Your project seems nice! I'll give it a try ;-) Only thing, the Jupiter-like part is not clear enough. 0: https://www.nurtch.com/ https://www.nurtch.com/ 1: https://docs.nurtch.com/en/latest/rubix-library/index.html https://docs.nurtch.com/en/latest/rubix-library/index.html 2: https://docs.gitlab.com/ee/user/project/clusters/runbooks/ https://docs.gitlab.com/ee/user/project/clusters/runbooks/