3 ms·

> which upon investigation might not even be “just” noise My company (like so many) is struggling a bit with culture around noisy alarm. Not only is noise tole

> which upon investigation might not even be “just” noise My company (like so many) is struggling a bit with culture around noisy alarm. Not only is noise tolerated, but when some closes an alarm because it's "known to be noise" and I prod them, it turns out that there is a very real impact on the user, it's just that nobody bothered to look into it. The alarm rings, the on-call hopes that it closes itself soon enough, it does so they consider it "false positive and noisy" even though there was impact on the user during these few minutes. The only way to fight that is a zero-tolerance culture on alarms, which means no false positive is ever tolerated: fix it.

stackskipton 2y ago
Ops person here, fixing it is generally impossible as most alarms are due to code and development teams don't give a shit. "It's just one false alarm occasionally, what's the big deal?" or "We are no longer working on that product." or "I have 14 features to do, go away ops and deal." If we completely shut off the alarm, one time it's actually an issue, I'll get dinged for shutting down monitoring. So yea, I mark it's priority as low so it doesn't wake me up in PagerDuty and move on.
bongodongobob 2y ago
A lot of times ops can't fix it (false positive), it's another teams responsibility. And if that other team can't or won't fix it, ops is screwed with constant false positives. At that point, it's in ops best interest to ignore it, let the actual positive wreak havoc and point at the other team. If they don't have the capability of putting pressure on that team, sometimes a fire is the only way to do it. I'm not saying this is a good idea, but bureaucracy is going to bureaucracy. You have to make the other team feel your pain.
aray07 2y ago
Yeah, I was surprised at how common this behavior has been across companies. I don't have a good solution for this problem - I am hoping additional visibility and accountability into these alerts can be useful. One of the things we have been hoping to do is to be able to add user impact context into the alert enrichment as well.

stackskipton 2y ago

Ops person here, fixing it is generally impossible as most alarms are due to code and development teams don't give a shit. "It's just one false alarm occasionally, what's the big deal?" or "We are no longer working on that product." or "I have 14 features to do, go away ops and deal." If we completely shut off the alarm, one time it's actually an issue, I'll get dinged for shutting down monitoring. So yea, I mark it's priority as low so it doesn't wake me up in PagerDuty and move on.

bongodongobob 2y ago

A lot of times ops can't fix it (false positive), it's another teams responsibility. And if that other team can't or won't fix it, ops is screwed with constant false positives. At that point, it's in ops best interest to ignore it, let the actual positive wreak havoc and point at the other team. If they don't have the capability of putting pressure on that team, sometimes a fire is the only way to do it. I'm not saying this is a good idea, but bureaucracy is going to bureaucracy. You have to make the other team feel your pain.

aray07 2y ago

Yeah, I was surprised at how common this behavior has been across companies. I don't have a good solution for this problem - I am hoping additional visibility and accountability into these alerts can be useful. One of the things we have been hoping to do is to be able to add user impact context into the alert enrichment as well.