3 ms·

I think you may be correct. >> We spent about 45 minutes looking at the alert catalog, the runbook for the services and trying to fix the underlying problem. A

I think you may be correct. >> We spent about 45 minutes looking at the alert catalog, the runbook for the services and trying to fix the underlying problem. After getting more information, we realized it was not affecting customers and could wait until the team that owned the service came online. Well, now, that is not a sev 1 alert. If you're on-call, you have the power to adjust alerts! Change it to a sev 2 so it does not alert future on-calls unneccessarily.

scottlamb 5y ago
> If you're on-call, you have the power to adjust alerts! Amen. Sounds like the author went even further: rather than just tweaking an alert, she deleted one, then deleted the whole service. That won't page again! > Change it to a sev 2 so it does not alert future on-calls unnecessarily. Or restructure the alert. I prefer SLO-oriented pages like "we've burned 1% of our quarterly error budget in the last hour" rather than cause-based pages like "CPU usage has exceeded 90% for 10 minutes". If you are getting paged based on something that doesn't affect customers, it might be a sign that you have the wrong alerts so you can't tell if they represent something important or not. Manually toggling the severity each time it's wrong might not be good enough. In terms of overall operational load though this week doesn't look bad at all: four surprise pages in seven days. After her fixes the next week should be lighter (knock on wood).