3 ms·

Did the recovery go well because of the single-env set-up, or despite of it?

• withinboredom 4y ago
  Well, the fact that we could just 'turn the feature off' was something that you'd generally only see because you /must/ deploy code that is disabled to keep PR's relatively small. So pretty much anything and everything can be entirely disabled -- it really depends on how long the feature has been enabled though. If it has been 'always on' for years, its possible new devs didn't include the check when modifying/adding the feature to new places. Then there was the fact that it was deleted at all. Once in 5 years, with a total dev team in the hundreds ... isn't that bad. They only pulled it off by doing something they were told not to do. That can happen anywhere. Also, since devops knew that it /could/ happen (because there is always someone who does what they aren't supposed to do -- including themselves when things break) took streaming backups of data. They were able to rebuild the table while the original was deleting and swap it back in as soon as the db released its write-lock. To me, I hadn't seen that level of 'fixing things' in prod. At 'regular' companies, if someone managed to delete a table, they'd probably spend at least twice that time finding someone who knew where the backups were and figuring out how to restore them. Meanwhile prod would probably be down completely since they can't disable the feature.