2 ms·
>The result? Shifting to this model from an SRE firefighting model has lowered total alerts, incidents, and outages. When we first shifted, people dreaded going
by fatnoah 7y ago
>The result? Shifting to this model from an SRE firefighting model has lowered total alerts, incidents, and outages. When we first shifted, people dreaded going on their on call rotation for their team; you expected to be paged multiple times a night.
I implemented a similar change at a previous company. Prior to the change, the operations team fronted all on-call issues. After, anything related to the software/services was handled by software teams. We went from nightly alerts to less than monthly. As a bonus, time to diagnose and repair went waayyyyy down due to much better logging.
Note, better logging <> more logging in this case. It meant less spurious errors/warnings, more useful INFO, and better observability of the system.