10 ms·

One of the best senior engineers I worked with taught me how to run an outage. The most important thing? Stop what you are doing, take charge, and get everyone

One of the best senior engineers I worked with taught me how to run an outage. The most important thing? Stop what you are doing, take charge, and get everyone else to stop what they are doing. The best case scenario of a bunch of engineers flailing about on a bridge turning knobs is that you luck into a fix but don't know how you got there. But you're more likely to make things worse.

londons_explore 5y ago
In my experience, the best strategy depends a lot on the severity of the outage. If all the alarms are going off because of a loss of redundancy, then currently there is no outage. The correct move should be carefully considered, and maybe tested in the sandbox environment. If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy.
- ComputerGuru 5y ago
  > If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy. Maybe. Once an outage has happened, an additional 30 minutes or hour of outage, depending 100% on the service in question, might be a bearable cost if it means preventing a domino effect of future issues caused by measures taken to restore the outage in a hurry.
- CGamesPlay 5y ago
  > If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy. Almost assuredly not. If a system hits a 100% outage, there are about to be a series of cascading failures by dependent systems. If you don't even understand which system is the root cause, all you're doing is testing a bunch of never-before-tried combinations in production and hoping something works.
boardwaalk 5y ago
Sounds like “locking the doors” (Space Shuttle disasters). Although, there was really not much to recover from there.
- etskinner 5y ago
  I hadn't heard of this before, chilling but cool: https://www.theguardian.com/world/2003/feb/13/columbia.spaceexploration https://www.theguardian.com/world/2003/feb/13/columbia.space...
sunstone 5y ago
I envy those innocent souls who have never heard of the concept of configuration management.

londons_explore 5y ago

In my experience, the best strategy depends a lot on the severity of the outage. If all the alarms are going off because of a loss of redundancy, then currently there is no outage. The correct move should be carefully considered, and maybe tested in the sandbox environment. If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy.

ComputerGuru 5y ago
> If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy. Maybe. Once an outage has happened, an additional 30 minutes or hour of outage, depending 100% on the service in question, might be a bearable cost if it means preventing a domino effect of future issues caused by measures taken to restore the outage in a hurry.
CGamesPlay 5y ago
> If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy. Almost assuredly not. If a system hits a 100% outage, there are about to be a series of cascading failures by dependent systems. If you don't even understand which system is the root cause, all you're doing is testing a bunch of never-before-tried combinations in production and hoping something works.

boardwaalk 5y ago

Sounds like “locking the doors” (Space Shuttle disasters). Although, there was really not much to recover from there.

etskinner 5y ago
I hadn't heard of this before, chilling but cool: https://www.theguardian.com/world/2003/feb/13/columbia.spaceexploration https://www.theguardian.com/world/2003/feb/13/columbia.space...

sunstone 5y ago

I envy those innocent souls who have never heard of the concept of configuration management.