3 ms·
I agree with your intent and desire, but the fact is that this problem _keeps happening,_ and we can't fix it by advocating "well, just do alerts better." Ther
by DanHulton 2y ago
I agree with your intent and desire, but the fact is that this problem _keeps happening,_ and we can't fix it by advocating "well, just do alerts better." There's a lot of cultural inertia at a lot of places that leads to creating too many, too low-signal alerts, and fixing that across an entire company -- or, hell, industry -- is a magnificently tall order.
However, installing a tool to specifically reign in those garbage noisy alerts is a potentially easy, significant win for the time and mental health of on-call engineers.
I mean, it sounds like you can then afterwards go in and identify the alerts that are just noise, and having that data means you can take action. Maybe contact the teams that are writing the noisiest alerts, or prepare some data-driven engineering standards for the company, whatever. But that still falls into "fix the culture", which is famously hard to do by fiat.
- steve1977 2y agoIf you work at a shitty place, focus your energy on leaving the shitty place.
- jgraettinger1 2y agoOr, it can be self-evident to all that today's work can deliver more benefit if focused elsewhere, instead of root-causing fickle alert flakes. Customers buy products and services, not alert hygiene.
- karamanolev 2y agoThe shittiness of a place is defined by a lot more, and more important, attributes than alert hygiene. Culture, pay, location, industry, leadership. If one leaves companies for (relatively) minor things like that, there are basically no companies left to work for.
- steve1977 2y agoAlert hygiene is a symptom of bad culture. Not the only one of course.