4 ms·

At Facebook we would simulate an entire datacenter disappearing. When we first started doing it the datacenter would be chosen months in advance so that teams

At Facebook we would simulate an entire datacenter disappearing. When we first started doing it the datacenter would be chosen months in advance so that teams would have plenty of time to ensure their services can run without that specific datacenter. When I left this year, the datacenter would be randomly chosen on the same day it would be cut off.

chasd00 4y ago
That's pretty cool and ideal practice for a software firm but in one of the reddit threads they're talking about mass quits/refusal to work of ground crew at Denver because of the weather. I wonder how you could ever prepare for that? Keep a backup, airport scoped, ground crew in the waiting room?? You can't really do hot spares for people without time to gear/train up and the weather event is so widespread I doubt there's enough spare SWA human capacity across the whole nation even if you had C130s on standby everywhere ready to take workers where they're needed most. From a national security perspective, situations like this is why the Marines exist right? Ensure a rapid response while the rest of the machine gets moving. I feel bad for everyone involved, those affected and those trying to figure out a solution.
- landemva 4y ago
  > I wonder how you could ever prepare for that? Management could consider how pay and performance programs can help ensure business continuity. HR and MBA xls wizards don't understand how to manage for business longevity.
  - dpkirchner 4y ago
    Pay might not really be enough. Maybe management could try to find folks to babysit kids/take care of parents trapped at home, freeing up workers to come fly. They'll certainly fail, but at least they'll understand the plight of their workers.
    - bsg75 4y ago
      Strangers paid on the lowest cost contract to leave with your kids?
      - brewdad 4y ago
        What's the big deal? You can always make more. /s
  - manigandham 4y ago
    Do you really think the MBA wizards can't figure out some basic pay issues? It seems you're discounting just how complex HR can be, especially in the face of exigent circumstances. No amount of bonuses will immediately staff up an entire terminal in the face of a massive snowstorm.
    - lotsofpulp 4y ago
      While they may be able to figure it out, optimizing pay to quality of life at work ratios to ensure long term employee retention and loyalty has certainly not been a priority.
    - landemva 4y ago
      > No amount of bonuses will immediately staff up an entire terminal That is right and thus I would engage line management to figure out business continuity. > in the face of a massive snowstorm. It is winter. The storm was tough but not exceptional for the season of winter. Denver did not report tremendous amounts of snow. Few if any MBAs can get on the ramp and look a line employee in the eye and lend a hand. The wfh keyboard warriors don't know blue collar and therefore are unable to figure this out. MBAs can figure out ways to game their pay. HR can recommend team building 'fun' and non-revenue standby seats, which have minimal value to those employees flying with school age children. Shareholders should demand senior management unemployment applications.
      - macintux 4y ago
        > The storm was tough but not exceptional for the season of winter. Sure seems like this was an exceptional storm. Widespread, deep cold reaching into Mexico, snow falling across much of the U.S., Buffalo hit with the most snow in 20 years, records set multiple locations.
        landemva 4y ago
        A couple decades of weather variation is not exceptional, though some will disagree with me when their bonus is yearly. I recall it was cold in Texas last winter. Weather is seasonal and varies. I wonder if board of directors will compare performance with other airlines.
- drdec 4y ago
  It doesn't seem that farfetched for an airline to run a drill where a given airport is assumed inoperable to see how the system reacts. The expectation shouldn't be the same as the data center failure but you can learn what you aren't doing well enough.
  - deleted 4y ago
    [deleted]
- KerrAvon 4y ago
  You can certainly handle it better than SWA is handling it. Maybe if they’d actually run a simulation where flights out of one or more cities had to be wholesale canceled, they would be. You can’t fix the situation for everyone, but you can avoid fucking up this badly.

chasd00 4y ago

That's pretty cool and ideal practice for a software firm but in one of the reddit threads they're talking about mass quits/refusal to work of ground crew at Denver because of the weather. I wonder how you could ever prepare for that? Keep a backup, airport scoped, ground crew in the waiting room?? You can't really do hot spares for people without time to gear/train up and the weather event is so widespread I doubt there's enough spare SWA human capacity across the whole nation even if you had C130s on standby everywhere ready to take workers where they're needed most. From a national security perspective, situations like this is why the Marines exist right? Ensure a rapid response while the rest of the machine gets moving. I feel bad for everyone involved, those affected and those trying to figure out a solution.

landemva 4y ago
> I wonder how you could ever prepare for that? Management could consider how pay and performance programs can help ensure business continuity. HR and MBA xls wizards don't understand how to manage for business longevity.
- dpkirchner 4y ago
  Pay might not really be enough. Maybe management could try to find folks to babysit kids/take care of parents trapped at home, freeing up workers to come fly. They'll certainly fail, but at least they'll understand the plight of their workers.
  - bsg75 4y ago
    Strangers paid on the lowest cost contract to leave with your kids?
    - brewdad 4y ago
      What's the big deal? You can always make more. /s
- manigandham 4y ago
  Do you really think the MBA wizards can't figure out some basic pay issues? It seems you're discounting just how complex HR can be, especially in the face of exigent circumstances. No amount of bonuses will immediately staff up an entire terminal in the face of a massive snowstorm.
  - lotsofpulp 4y ago
    While they may be able to figure it out, optimizing pay to quality of life at work ratios to ensure long term employee retention and loyalty has certainly not been a priority.
  - landemva 4y ago
    > No amount of bonuses will immediately staff up an entire terminal That is right and thus I would engage line management to figure out business continuity. > in the face of a massive snowstorm. It is winter. The storm was tough but not exceptional for the season of winter. Denver did not report tremendous amounts of snow. Few if any MBAs can get on the ramp and look a line employee in the eye and lend a hand. The wfh keyboard warriors don't know blue collar and therefore are unable to figure this out. MBAs can figure out ways to game their pay. HR can recommend team building 'fun' and non-revenue standby seats, which have minimal value to those employees flying with school age children. Shareholders should demand senior management unemployment applications.
    - macintux 4y ago
      > The storm was tough but not exceptional for the season of winter. Sure seems like this was an exceptional storm. Widespread, deep cold reaching into Mexico, snow falling across much of the U.S., Buffalo hit with the most snow in 20 years, records set multiple locations.
      - landemva 4y ago
        A couple decades of weather variation is not exceptional, though some will disagree with me when their bonus is yearly. I recall it was cold in Texas last winter. Weather is seasonal and varies. I wonder if board of directors will compare performance with other airlines.
drdec 4y ago
It doesn't seem that farfetched for an airline to run a drill where a given airport is assumed inoperable to see how the system reacts. The expectation shouldn't be the same as the data center failure but you can learn what you aren't doing well enough.
- deleted 4y ago
  [deleted]
KerrAvon 4y ago
You can certainly handle it better than SWA is handling it. Maybe if they’d actually run a simulation where flights out of one or more cities had to be wholesale canceled, they would be. You can’t fix the situation for everyone, but you can avoid fucking up this badly.