3 ms·

Yes, and even more importantly: compensate your engineers who minimize the need for these kinds of heroics. Ones who: * fix the alerts so they reliably page wh

Yes, and even more importantly: compensate your engineers who minimize the need for these kinds of heroics. Ones who: * fix the alerts so they reliably page when there's a SLO-worthy problem and only then. * test the restore system so it works smoothly when needed at the necessary scale. * add safety checks to prevent the need to use those backups in the first place. * get to the root cause of yesterday's outage and prioritize the 9–5 engineering work to ensure it won't happen again. It's awful to work for a place where heroics are often necessary and unrewarded. I still don't like working in a place where heroics are often necessary, even if they're celebrated. They're often avoidable.

morelisp 4y ago
The trick is to make sure the heroic efforts are rewarded quietly, but the unheroic ones create lots of noise. This is inherently difficult because the heroic efforts tend to come as a result of very noisy incidents - everyone already knows the database cluster was literally on fire and wants to hear how Jane put it out while simultaneously remagnetizing the backup drives. The lead's job is to make the noise ensuring everyone knows Morgan's the one who did the boring rewiring of the backup-restore process to automatically sync and fail over to another AZ when thermal metrics start trending bad. One thing I've noticed as we've been trying to integrate some on-call processes across formerly separate teams is that "our" incident reports were about 25% timeline/responses/impact analysis and 75% RCA and plans for future mitigations. "Theirs" were the opposite. Our incident rate has trended down sharply over the past two years even as our system grew; theirs has scaled up roughly at the same rate as their service count.
aledalgrande 4y ago
and document anything and everything and write amazing runbooks
maerF0x0 4y ago
> They're often avoidable. Avoidance isn't the goal of mgmt though. Cost minimization is, and by quantizing the compensation portion it can help figure out the cost of a bug. Ultimately the cost of a bug is a key variable in the decision to prevent or solve. The cost can easily vary from $0 (eg at a startup with no customers) or to millions (maybe billions?) when you consider the risk of brand damage, lost ARR (multiply that ARR by the sales multiple!), decremented velocity (lost opportunity value), and eventually even recruiting cost (to replace frustrated employees). I've always wanted to work at a JoelTest[1] company which fixes bugs before writing new code... But the closest i've found is a commitment that the oncall engineer works on bugs for their rotation to reduce the bug backlog ( and is not part of sprint velocity) [1]: https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
- scottlamb 4y ago
  > Avoidance isn't the goal of mgmt though. Management's goals are easier to achieve when they can employ excellent engineers. As I explained, I don't want to work somewhere heroics are often needed. I'm not alone. That's a factor they need to consider, in addition to the customer impact of the outages themselves.
- sam0x17 4y ago
  Even at a startup with zero customers the cost of a bug is non-zero if you plan to allocate dev time to it or you have to fix it to be able to launch.
- throw0101a 4y ago
  > Avoidance isn't the goal of mgmt though. Cost minimization is […] How about (net) value creation? Whose goal is that? Minimize costs all you want, but if your service is garbage you won't be getting (as) much income either.

morelisp 4y ago

The trick is to make sure the heroic efforts are rewarded quietly, but the unheroic ones create lots of noise. This is inherently difficult because the heroic efforts tend to come as a result of very noisy incidents - everyone already knows the database cluster was literally on fire and wants to hear how Jane put it out while simultaneously remagnetizing the backup drives. The lead's job is to make the noise ensuring everyone knows Morgan's the one who did the boring rewiring of the backup-restore process to automatically sync and fail over to another AZ when thermal metrics start trending bad. One thing I've noticed as we've been trying to integrate some on-call processes across formerly separate teams is that "our" incident reports were about 25% timeline/responses/impact analysis and 75% RCA and plans for future mitigations. "Theirs" were the opposite. Our incident rate has trended down sharply over the past two years even as our system grew; theirs has scaled up roughly at the same rate as their service count.

aledalgrande 4y ago

and document anything and everything and write amazing runbooks

maerF0x0 4y ago

> They're often avoidable. Avoidance isn't the goal of mgmt though. Cost minimization is, and by quantizing the compensation portion it can help figure out the cost of a bug. Ultimately the cost of a bug is a key variable in the decision to prevent or solve. The cost can easily vary from $0 (eg at a startup with no customers) or to millions (maybe billions?) when you consider the risk of brand damage, lost ARR (multiply that ARR by the sales multiple!), decremented velocity (lost opportunity value), and eventually even recruiting cost (to replace frustrated employees). I've always wanted to work at a JoelTest[1] company which fixes bugs before writing new code... But the closest i've found is a commitment that the oncall engineer works on bugs for their rotation to reduce the bug backlog ( and is not part of sprint velocity) [1]: https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...

scottlamb 4y ago
> Avoidance isn't the goal of mgmt though. Management's goals are easier to achieve when they can employ excellent engineers. As I explained, I don't want to work somewhere heroics are often needed. I'm not alone. That's a factor they need to consider, in addition to the customer impact of the outages themselves.
sam0x17 4y ago
Even at a startup with zero customers the cost of a bug is non-zero if you plan to allocate dev time to it or you have to fix it to be able to launch.
throw0101a 4y ago
> Avoidance isn't the goal of mgmt though. Cost minimization is […] How about (net) value creation? Whose goal is that? Minimize costs all you want, but if your service is garbage you won't be getting (as) much income either.