5 ms·

It is never that simple. Say the backup existed, and was global. By the time you get everyone briefed on how fubar it is and get agreement to load the backups,

It is never that simple. Say the backup existed, and was global. By the time you get everyone briefed on how fubar it is and get agreement to load the backups, there are hours of changes from the unaffected customers that will be wiped by the restore, or have to be reconciled by hand for months. Sure, you can concoct the perfect antidote with hindsight, but their retro and next steps are sound.

rob_c 4y ago
Build it properly. By definiton of following best practice IT IS or always should be not far from being able to follow this. If its not someone is to blame
kwertyoowiyop 4y ago
That lesson really stuck out for me also. My definition of “restore” has been too simplistic.
- rob_c 4y ago
  Not really. They managed closer to "undo" the break, which still makes it look like their disaster recovery might also be un-tested. I'm sure there are many companies who would lose 24hr of tracking to get back online in a few hr vs 14days. In the real world with deadlines and commitments this is madness.
baskethead 4y ago
What it means is that they never tested their disaster recovery system, because this would have been found right away. Or, someone would have reported it and an upper level exec would have signed off on it being okay to take 14 days to restore a small subset of users.
- largbae 4y ago
  Again, not that simple. The customer restore procedure was almost certainly tested(and in active use as customers blow up their own data often enough). It was _not_ tested on 800 customer stacks simultaneously, as that was considered a sitewide disaster by whoever dreamed up the failure modes to test for. Meanwhile the actual whole site disaster restore plan may or may not have been tested, but it was useless for this case since some customers were unaffected and would be damaged by the whole site plan.
  - baskethead 4y ago
    Then we are in agreement. Even if it were the case they had to do 800 separate recoveries (which I vaguely remember reading they couldn't do it individually), it means they never tested a large scale recovery situation and had no idea that 800 recoveries would take 2 weeks. That's a significant issue that they should have tested.

rob_c 4y ago

Build it properly. By definiton of following best practice IT IS or always should be not far from being able to follow this. If its not someone is to blame

kwertyoowiyop 4y ago

That lesson really stuck out for me also. My definition of “restore” has been too simplistic.

rob_c 4y ago
Not really. They managed closer to "undo" the break, which still makes it look like their disaster recovery might also be un-tested. I'm sure there are many companies who would lose 24hr of tracking to get back online in a few hr vs 14days. In the real world with deadlines and commitments this is madness.

baskethead 4y ago

What it means is that they never tested their disaster recovery system, because this would have been found right away. Or, someone would have reported it and an upper level exec would have signed off on it being okay to take 14 days to restore a small subset of users.

largbae 4y ago
Again, not that simple. The customer restore procedure was almost certainly tested(and in active use as customers blow up their own data often enough). It was _not_ tested on 800 customer stacks simultaneously, as that was considered a sitewide disaster by whoever dreamed up the failure modes to test for. Meanwhile the actual whole site disaster restore plan may or may not have been tested, but it was useless for this case since some customers were unaffected and would be damaged by the whole site plan.
- baskethead 4y ago
  Then we are in agreement. Even if it were the case they had to do 800 separate recoveries (which I vaguely remember reading they couldn't do it individually), it means they never tested a large scale recovery situation and had no idea that 800 recoveries would take 2 weeks. That's a significant issue that they should have tested.