3 ms·
I suppose it all depends on how much infra needs to be stood up for the absolute necessities of the business to operate. Does the company need that internal ti
by wingmanjd 5y ago
I suppose it all depends on how much infra needs to be stood up for the absolute necessities of the business to operate. Does the company need that internal ticketing system in place to process external client transactions? Probably not, but it'll need it eventually (so maybe that moves to 2nd tier restore process?). My company's RTO is 24hrs to processing new client transactions. Restoring old ones will definitely take longer, but at least new ones can proceed.
If your own company's RTO is 2w, that sounds like a lot needs to be in place. Part of the business continuity/ disaster recovery is getting management to sign off on those types of numbers, big or small. Make sure they're realistic.
You're right that this type of recovery is not fun. Bryan Cantrill gives a great presentation about managing an outage (https://www.youtube.com/watch?v=30jNsCVLpAE https://www.youtube.com/watch?v=30jNsCVLpAE). One of my biggest takeaways, if you're looking at a sweeping outage and a long haul of a recovery, do sleep management asap with your team. Dead tired people are more likely to make brain dead decisions.
- bogomipz 5y agoWhat a great link, thanks for sharing. >"Dead tired people are more likely to make brain dead decisions" Indeed. In reading the post-mortem on the recent multi-day Roblox outage, it's hard not to imagine some bad decisions were made that only made problems worse and that likely these were on account of people just being fried by lack of sleep: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ https://blog.roblox.com/2022/01/roblox-return-to-service-10-...