3 ms·
> If you're on-call, you have the power to adjust alerts! Amen. Sounds like the author went even further: rather than just tweaking an alert, she deleted one,
by scottlamb 5y ago
> If you're on-call, you have the power to adjust alerts!
Amen.
Sounds like the author went even further: rather than just tweaking an alert, she deleted one, then deleted the whole service. That won't page again!
> Change it to a sev 2 so it does not alert future on-calls unnecessarily.
Or restructure the alert. I prefer SLO-oriented pages like "we've burned 1% of our quarterly error budget in the last hour" rather than cause-based pages like "CPU usage has exceeded 90% for 10 minutes". If you are getting paged based on something that doesn't affect customers, it might be a sign that you have the wrong alerts so you can't tell if they represent something important or not. Manually toggling the severity each time it's wrong might not be good enough.
In terms of overall operational load though this week doesn't look bad at all: four surprise pages in seven days. After her fixes the next week should be lighter (knock on wood).