3 ms·
I work on a team which runs hyper critical infra on all production machines at BigCo and have the same experience as you. The problem are not the alerts — the
by __turbobrew__ 2y ago
I work on a team which runs hyper critical infra on all production machines at BigCo and have the same experience as you.
The problem are not the alerts — the alerts actually are catching real problems — the problem is the following:
1. The team is understaffed so sometimes spending a few days root causing an alert is not prioritized
2. When alerts are root caused sometimes the work to fix the root cause is not prioritized
3. A culture on the team which allows alerts to go untriaged due to desensitization.
Our headcount got reduced by ~40% and — surprise surprise — reliability and on-call got much worse. Senior leadership has made the decision that the cost cuts are worth the decreased reliability so nothing is going to change.
The job market is rough so people put up with this for now.
- kbar13 2y agowe as an industry need to have engineering management types realize that we cannot prioritize roadmap to the complete detriment of reliability
- cdchn 2y agoIf your org claims to be "customer obsessed" then reframe your alerts as what their impact to customers are. Don't say "elevated 502 errors" say "customers couldn't encountered errors X times."
- lupire 2y agoIt sounds like you forget to make an SLO? If an alert is not actionable because it's impossible to resolve, even though it has customer impact, then it should be an SLO, not an alert.
- lazyasciiart 2y agoSLO as in “service level objective”? How does defining an SLO stop the existence of alerts?
- mike_d 2y agoStart putting together conference bridges for "P1 customer outages" and have someone who is responsible for calling the developers, PMs, scrum masters, managers, etc. on the team and getting them all on at 1 AM to fix it.
- callalex 2y agoWhen describing infrastructure, words matter. When you describe something as “hyper critical infrastructure” it implies that tens to thousands of human beings will die within seconds of failure of said “hyper critical” infrastructure. The way the rest of your comment is worded implies that’s not what you’re actually describing and makes the words “hyper critical infrastructure” irresponsible for you to use. I don’t mean to imply there is some kind of failure magnitude competition, I just want to reinforce that software “engineering” already has a huge problem with abject neglect of the learnings that other sign-and-stamp engineering fields have already learned from and fixed. Us code slingers are not in uncharted territory, we just need to learn from our predecessors and peers that build literal bridges and towers and force management to treat our field in the same way.
- dambi0 2y agoWords matter but so does context. You weren’t confused by the words here why assume others would be?
- __turbobrew__ 2y agoHyper critical means if it stop working potentially billions of dollars are lost for the employees and shareholders. Given that the FAA values a human life at $9 million that actually fits your arbitrary criteria of what I am allowed to call my job.