3 ms·

Not OP, but at a previous company we were given access to a new AWS environment with nothing in it and were timed on how quickly we could get all of our service

Not OP, but at a previous company we were given access to a new AWS environment with nothing in it and were timed on how quickly we could get all of our services operational. Fleet teams went first, then networking, then T1 services (identity, platform, etc) and so on

lanstin 5y ago
So roughly how long did it take? I participated in table exercises of that sort and couldn’t convincingly get less than two weeks. And not two fun weeks.
- wingmanjd 5y ago
  I suppose it all depends on how much infra needs to be stood up for the absolute necessities of the business to operate. Does the company need that internal ticketing system in place to process external client transactions? Probably not, but it'll need it eventually (so maybe that moves to 2nd tier restore process?). My company's RTO is 24hrs to processing new client transactions. Restoring old ones will definitely take longer, but at least new ones can proceed. If your own company's RTO is 2w, that sounds like a lot needs to be in place. Part of the business continuity/ disaster recovery is getting management to sign off on those types of numbers, big or small. Make sure they're realistic. You're right that this type of recovery is not fun. Bryan Cantrill gives a great presentation about managing an outage (https://www.youtube.com/watch?v=30jNsCVLpAE https://www.youtube.com/watch?v=30jNsCVLpAE). One of my biggest takeaways, if you're looking at a sweeping outage and a long haul of a recovery, do sleep management asap with your team. Dead tired people are more likely to make brain dead decisions.
  - bogomipz 5y ago
    What a great link, thanks for sharing. >"Dead tired people are more likely to make brain dead decisions" Indeed. In reading the post-mortem on the recent multi-day Roblox outage, it's hard not to imagine some bad decisions were made that only made problems worse and that likely these were on account of people just being fried by lack of sleep: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ https://blog.roblox.com/2022/01/roblox-return-to-service-10-...
- babelfish 5y ago
  The PM in charge of it spent probably 6 months to a year planning it with every team prior to our first gameday. The first gameday took 2-5 days to complete entirely, but we made it a bi-annual exercise and eventually got it down to ~12 hours before I left the company
bogomipz 5y ago
I'm curious why Fleet teams went before networking? Shouldn't the dependency there require it to be the other way around?
- babelfish 5y ago
  This was a couple years ago, you may be right, sorry for the confusion
  - wingmanjd 5y ago
    What is a fleet team? I'm not familiar with that term.
    - babelfish 5y ago
      Company was in the middle of an on-prem -> cloud transition. They were responsible for the on-prem hardware as well as provisioning VMs across cloud instances

lanstin 5y ago

So roughly how long did it take? I participated in table exercises of that sort and couldn’t convincingly get less than two weeks. And not two fun weeks.

wingmanjd 5y ago
I suppose it all depends on how much infra needs to be stood up for the absolute necessities of the business to operate. Does the company need that internal ticketing system in place to process external client transactions? Probably not, but it'll need it eventually (so maybe that moves to 2nd tier restore process?). My company's RTO is 24hrs to processing new client transactions. Restoring old ones will definitely take longer, but at least new ones can proceed. If your own company's RTO is 2w, that sounds like a lot needs to be in place. Part of the business continuity/ disaster recovery is getting management to sign off on those types of numbers, big or small. Make sure they're realistic. You're right that this type of recovery is not fun. Bryan Cantrill gives a great presentation about managing an outage (https://www.youtube.com/watch?v=30jNsCVLpAE https://www.youtube.com/watch?v=30jNsCVLpAE). One of my biggest takeaways, if you're looking at a sweeping outage and a long haul of a recovery, do sleep management asap with your team. Dead tired people are more likely to make brain dead decisions.
- bogomipz 5y ago
  What a great link, thanks for sharing. >"Dead tired people are more likely to make brain dead decisions" Indeed. In reading the post-mortem on the recent multi-day Roblox outage, it's hard not to imagine some bad decisions were made that only made problems worse and that likely these were on account of people just being fried by lack of sleep: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ https://blog.roblox.com/2022/01/roblox-return-to-service-10-...
babelfish 5y ago
The PM in charge of it spent probably 6 months to a year planning it with every team prior to our first gameday. The first gameday took 2-5 days to complete entirely, but we made it a bi-annual exercise and eventually got it down to ~12 hours before I left the company

bogomipz 5y ago

I'm curious why Fleet teams went before networking? Shouldn't the dependency there require it to be the other way around?

babelfish 5y ago
This was a couple years ago, you may be right, sorry for the confusion
- wingmanjd 5y ago
  What is a fleet team? I'm not familiar with that term.
  - babelfish 5y ago
    Company was in the middle of an on-prem -> cloud transition. They were responsible for the on-prem hardware as well as provisioning VMs across cloud instances