3 ms·
This tool looks great - it automates a lot of the process we have for major incidents at my company! One feature I was hoping this would have, related to on-ca
by zinodaur 4y ago
This tool looks great - it automates a lot of the process we have for major incidents at my company!
One feature I was hoping this would have, related to on-call, but for minor week-to-week incidents:
My team gets many low priority alerts every week that need to be triaged, worked on, or ignored (something on the scale of 10 - 20 alerts per week, often re-occurring, often firing with slightly different tags).
These alerts are "symptoms" of something wrong with the system, and the oncall always has to manually sort them into "causes". A "cause" is then tracked and worked on, with investigation notes etc.
I've seen this workflow at a number of companies, and it is always handled with poor automation or none. Oncall engineers manually copy-paste the pagerduty - alert url into a spreadsheet (or google doc), manually assign a status tag, manually group re-occurring alerts into the same row, manually assign engineers (and the assigned engineers have to manually ack being assigned!).
Most of the time, the "symptom-grouping" decisions have to be made by a human, since a lot of the time it requires investigation to determine the link between two symptoms. But just because that one decision has to be made by a human, doesn't mean that all the bookkeeping shouldn't be automated!
With all these manual steps, there are so many places for human error.
We would pay so much money for a tool that did this for us
- tyingq 4y agoThe industry terms are "incident management" (symptoms) and "problem management" (root causes), which might be helpful searching for existing tools.