3 ms·

Yeah feels like there's a lot of room for startups in this space (Blameless has a great demo, too). No one wants to rewrite PagerDuty internally -- why are pe

Yeah feels like there's a lot of room for startups in this space (Blameless has a great demo, too). No one wants to rewrite PagerDuty internally -- why are people all writing their bespoke incident management, response, review, and reporting toolchains internally too?

jedberg 7y ago
> why are people all writing their bespoke incident management, response, review, and reporting toolchains internally too? Because the good ones are tightly integrated with the rest of the internal tooling. To use a 3rd party incident management tool usually means you have to run your operations they way they expect to get maximum value. A lot of times its easier to build it yourself based on how you operate. However, as the way people operate become more standardized, the third party tools will become more useful.
- mherdeg 7y ago
  Yeah, agree, this is probably the key sticking point: the 3rd party toolchain doesn't do [some key thing] the stakeholders in my process have found valuable. So I end up run some scenario analysis and find that the 3rd party toolchain doesn't shave off enough developer time & ongoing support cost from the rest of the operational work I'm going to have to do anyway to manage an incident. And I load the vendor's web page every couple of months and wistfully wish my process were uncomplicated enough to use their tools. Here's hoping that the calculus changes either as these tools grow more robust or, like you say, as people begin to manage their software systems in less bespoke ways. For example, during an incident we'd like to know "what changed between time [X] and [Y]" (deployments, configuration, experiments, other service outages) while we're trying to determine a root cause and fix the problem. And much later, after the incident, we'd like to auto-compute a metric like "what is the change success rate of [Y service / services owned by team Z]?". This aren't complicated concepts -- it's not like we're trying to reason about causality with machine learning to reduce time-to-resolve for incidents! Still, our incident-management tool will really behave better if it's aware of our what-changed-at-time-X tooling and our incidents-caused-by-changes reporting. If this is an external tool, yikes, now our incident tool has sprung awareness of changes and reporting?!
quartz 7y ago
For a lot of companies I think the internal processes and tools for major incidents have been sort of cobbled together over time. It's somewhat recent that organizations (especially smaller companies) are thinking about major incidents more diligently as an end-to-end process that's connected to but separate from more general ops-alerts. One of the reasons we started Kintaba was that we're noticing a shift as more and more employees are leaving successful companies that have invested tons of time into their incident management process (most of the bigtech companies) and are now joining/starting their own companies where they want to spin up a mature process quickly.

jedberg 7y ago

> why are people all writing their bespoke incident management, response, review, and reporting toolchains internally too? Because the good ones are tightly integrated with the rest of the internal tooling. To use a 3rd party incident management tool usually means you have to run your operations they way they expect to get maximum value. A lot of times its easier to build it yourself based on how you operate. However, as the way people operate become more standardized, the third party tools will become more useful.

mherdeg 7y ago
Yeah, agree, this is probably the key sticking point: the 3rd party toolchain doesn't do [some key thing] the stakeholders in my process have found valuable. So I end up run some scenario analysis and find that the 3rd party toolchain doesn't shave off enough developer time & ongoing support cost from the rest of the operational work I'm going to have to do anyway to manage an incident. And I load the vendor's web page every couple of months and wistfully wish my process were uncomplicated enough to use their tools. Here's hoping that the calculus changes either as these tools grow more robust or, like you say, as people begin to manage their software systems in less bespoke ways. For example, during an incident we'd like to know "what changed between time [X] and [Y]" (deployments, configuration, experiments, other service outages) while we're trying to determine a root cause and fix the problem. And much later, after the incident, we'd like to auto-compute a metric like "what is the change success rate of [Y service / services owned by team Z]?". This aren't complicated concepts -- it's not like we're trying to reason about causality with machine learning to reduce time-to-resolve for incidents! Still, our incident-management tool will really behave better if it's aware of our what-changed-at-time-X tooling and our incidents-caused-by-changes reporting. If this is an external tool, yikes, now our incident tool has sprung awareness of changes and reporting?!

quartz 7y ago

For a lot of companies I think the internal processes and tools for major incidents have been sort of cobbled together over time. It's somewhat recent that organizations (especially smaller companies) are thinking about major incidents more diligently as an end-to-end process that's connected to but separate from more general ops-alerts. One of the reasons we started Kintaba was that we're noticing a shift as more and more employees are leaving successful companies that have invested tons of time into their incident management process (most of the bigtech companies) and are now joining/starting their own companies where they want to spin up a mature process quickly.