4 ms·
Yeah, our last on call incident was like 2 weeks ago. Before that, months ago. A long time ago we had every little thing page us. Plz no. So many false hits.
by sbov 8y ago
Yeah, our last on call incident was like 2 weeks ago. Before that, months ago.
A long time ago we had every little thing page us. Plz no. So many false hits.
Things that can wait don't page us. It sends an email, and if we see it and have nothing else better to do, we might fix it. Otherwise it waits till work hours. Paging means something needs to be fixed now.
I have a friend who gets paged even for "low" (not really low) disk space. Here's a tip: turn up the alerts during work hours so you can resolve that junk before people on call have to deal with it.
- bigiain 8y agoThe linked page on Alerting Principle there is good: https://response.pagerduty.com/oncall/alerting_principles/ https://response.pagerduty.com/oncall/alerting_principles/ "We manage how we get alerted based on a simple principle, an alert is something which requires a human to perform an action. Anything else is a notification, which is something that we cannot control, and for which we cannot make any action to affect it. Notifications are useful, but they shouldn't be waking people up under any circumstance." They also distinguish between "needs fixing right now", and "needs to be fixed asap, but can wait until office hours"...
- outworlder 8y ago> I have a friend who gets paged even for "low" (not really low) disk space. I am your friend, I guess. Fighting this battle for a couple of years now. Change is hard. I keep saying that I couldn't care less if CPU has reached 100% in some random server, if customer metrics are unaffected. It is not an emergency. Same for disk. I don't care if it is at 80% full. Page me if it is going to run out before the weekend ends. Or in a month if it's during normal business hours. Were it up to me, I'd tear down all these metrics and replace them with the four golden signals.