4 ms·

Similar to Facebook's Storm initiative: https://www.forbes.com/sites/roberthof/2016/09/11/interview-how-facebooks-project-storm-heads-off-data-center-disasters/

• Shish2k 4y ago
  I used to be heavily involved in those, it was a process where we’d take a week to prepare for it, do weeks of post-mortems, and print a run of t-shirts for everyone involved to celebrate pulling one off successfully. These days the team running them announces that it’s happening in an opt-in announcement group at 8am, pulls the plug at 9am, and barely anyone even notices because the automation handles it so gracefully. Mostly I just miss the t-shirts, as the <datacenter>-storm events got the coolest graphical designs...
  • wrigby 4y ago
    Oh hey there! Can confirm that Storms got very very smooth over the course of a few years - it was really incredible to see it go from something that we planned on our roadmap to getting into work and realizing an entire datacenter was disconnected from the backbone while everything ran mostly smooth.