4 ms·

I've noticed a lack of of human planning in most DR plans. In some industries it's called business continuance. Our company has focused mainly on the technical

by yardie 12y ago

I've noticed a lack of of human planning in most DR plans. In some industries it's called business continuance. Our company has focused mainly on the technical side of the business: backups, DR, offsite access. Depending on the situation, earthquakes can bring fires and tsunamis. So what do you tell the network engineer that had their house burn to the ground? "Sorry for your lose; we're decamping for Las Vegas. Wheels up in 8 hours"? If the big one does hit there is a lot more at stake than just your precious data.

ObviousScience 12y ago
I imagine for a large tech company, it would go something like this: 1. Immediately call extra hands in other locations (datacenters, customer service, etc); begin over-time scheduling to alleviate need for workers in disaster zone. Doesn't require any coordination of disaster zone, just remaining executives located outside. 2. Offer two special payments to workers in the impacted zone: 1. Disaster recovery assistance, to help them cushion emergency needs and get their family/belongings/etc recovered faster; 2. Relocation bonus, to create an incentive for quick relocation to temporary (or new) office. 3. Spend as much as you to have to get extra emergency services in to the area, and generally assist in government/regional recovery area. This will free up the last of your staff for temporary relocation. I doubt that a large company (we're talking Google, Microsoft, Facebook, etc) would take more than a couple days to have other sites working overtime to cover the load, less than a week to have already transported some of their staff to a new site, and less than a month to have brought almost all of the office back online. This timeline, of course, assumes that the problem wasn't that their office fell on all their employees, killing them or something of that nature. Likely, we'd see a hiccup in the development cycle, but in terms of keeping infrastructure online, emergency bug fixes, etc, we'd probably see relatively little damage.
- yardie 12y ago
  Disasters tend to have a knock on effect that can ripple for years. 2 that come to mind were the floods in Thailand and the 9-11 terrorist act. The floods temporarily knocked out the supply chain of HDD components, which had the knock on of system builders not having enough components. And eventually, technicians not having enough spare replacements for current systems. 1. In most businesses these other locations are supplementary to HQ. Typically there is no one at the datacenter and CS centers rely on guidance from HQ. They are not in the position to make decisions and drive the company. 2. Growing up and working in Florida you natural disasters are like clockwork. I can say for sure that 1 is covered by insurance and the gov't and 2 is going to be hard to pull off without looking like a complete dick. Also, you have other constraints. Airports may be closed and driving can be dangerous. Plus cars need fuel which is typically in short supply and expensive. 3. If you have a company of a few thousand I would say a few hundred are essential. Where are you going to find a place to house a few hundred people at once? Remember that other companies are thinking the same as well. I also believe the most essential employees are that way for a reason. If they have the same preparedness in the company they also have the same mentality in their community. It would not be illogical to imagine the person in charge of DR for the company is also a member of the volunteer fire department. Microsoft would be able to recover, then they find out the caterers don't have food to serve the cafeteria, employees can't come to work because there is no fuel, etc. These are the soft issues that most DR plans don't account for.
manifesto 12y ago
Google has a program called DiRT (Disaster Recovery Testing)[0], when in a certain week of the year, some major exercises that simulating the loss of headquarters are carried out, including the (simulated) loss of some teams, some network connections in the headquarter and/or the physical access to the headquarters buildings. That makes a lot of sense when you place your headquarters at a place that can be demolished by an earthquake. [0]: http://queue.acm.org/detail.cfm?id=2371516 http://queue.acm.org/detail.cfm?id=2371516
- yardie 12y ago
  That is actually an excellent idea. Loss of manpower is something that is overlooked in a lot of companies. I believe part of it is job security (make yourself invaluable). And part of it is HR optimisation. Why pay for 2 people to do the same job when 1 will do? The same reason you pay for 2 servers, 2 backup processes, etc.
quattrofan 12y ago
Not a case of "if" but when.
laurenstill 12y ago
Used to work in Florida, and it's a pretty noticeable difference in DR/Continuity strategy to what I work with now out here (SF). For the first 24 hours after Hurricane Ivan it was completely focused on: Are our employees safe; do they have water/food; what medical aid is needed; how can we get blankets and cots into our office so our employees have a place to sleep tonight? All the roads are washed out, there was no real in/out of town for a week, no power to our main DC for 3. Having redundancy in Chicago and Virginia didn't really mean shit when we couldn't get key people to those data centers to run them full steam. The end result was a high reliance on contract employees at the fail-overs, and a real appreciation for the human element.

by yardie 12y ago

ObviousScience 12y ago

I imagine for a large tech company, it would go something like this: 1. Immediately call extra hands in other locations (datacenters, customer service, etc); begin over-time scheduling to alleviate need for workers in disaster zone. Doesn't require any coordination of disaster zone, just remaining executives located outside. 2. Offer two special payments to workers in the impacted zone: 1. Disaster recovery assistance, to help them cushion emergency needs and get their family/belongings/etc recovered faster; 2. Relocation bonus, to create an incentive for quick relocation to temporary (or new) office. 3. Spend as much as you to have to get extra emergency services in to the area, and generally assist in government/regional recovery area. This will free up the last of your staff for temporary relocation. I doubt that a large company (we're talking Google, Microsoft, Facebook, etc) would take more than a couple days to have other sites working overtime to cover the load, less than a week to have already transported some of their staff to a new site, and less than a month to have brought almost all of the office back online. This timeline, of course, assumes that the problem wasn't that their office fell on all their employees, killing them or something of that nature. Likely, we'd see a hiccup in the development cycle, but in terms of keeping infrastructure online, emergency bug fixes, etc, we'd probably see relatively little damage.

yardie 12y ago
Disasters tend to have a knock on effect that can ripple for years. 2 that come to mind were the floods in Thailand and the 9-11 terrorist act. The floods temporarily knocked out the supply chain of HDD components, which had the knock on of system builders not having enough components. And eventually, technicians not having enough spare replacements for current systems. 1. In most businesses these other locations are supplementary to HQ. Typically there is no one at the datacenter and CS centers rely on guidance from HQ. They are not in the position to make decisions and drive the company. 2. Growing up and working in Florida you natural disasters are like clockwork. I can say for sure that 1 is covered by insurance and the gov't and 2 is going to be hard to pull off without looking like a complete dick. Also, you have other constraints. Airports may be closed and driving can be dangerous. Plus cars need fuel which is typically in short supply and expensive. 3. If you have a company of a few thousand I would say a few hundred are essential. Where are you going to find a place to house a few hundred people at once? Remember that other companies are thinking the same as well. I also believe the most essential employees are that way for a reason. If they have the same preparedness in the company they also have the same mentality in their community. It would not be illogical to imagine the person in charge of DR for the company is also a member of the volunteer fire department. Microsoft would be able to recover, then they find out the caterers don't have food to serve the cafeteria, employees can't come to work because there is no fuel, etc. These are the soft issues that most DR plans don't account for.

manifesto 12y ago

Google has a program called DiRT (Disaster Recovery Testing)[0], when in a certain week of the year, some major exercises that simulating the loss of headquarters are carried out, including the (simulated) loss of some teams, some network connections in the headquarter and/or the physical access to the headquarters buildings. That makes a lot of sense when you place your headquarters at a place that can be demolished by an earthquake. [0]: http://queue.acm.org/detail.cfm?id=2371516 http://queue.acm.org/detail.cfm?id=2371516

yardie 12y ago
That is actually an excellent idea. Loss of manpower is something that is overlooked in a lot of companies. I believe part of it is job security (make yourself invaluable). And part of it is HR optimisation. Why pay for 2 people to do the same job when 1 will do? The same reason you pay for 2 servers, 2 backup processes, etc.

quattrofan 12y ago

Not a case of "if" but when.

laurenstill 12y ago

Used to work in Florida, and it's a pretty noticeable difference in DR/Continuity strategy to what I work with now out here (SF). For the first 24 hours after Hurricane Ivan it was completely focused on: Are our employees safe; do they have water/food; what medical aid is needed; how can we get blankets and cots into our office so our employees have a place to sleep tonight? All the roads are washed out, there was no real in/out of town for a week, no power to our main DC for 3. Having redundancy in Chicago and Virginia didn't really mean shit when we couldn't get key people to those data centers to run them full steam. The end result was a high reliance on contract employees at the fail-overs, and a real appreciation for the human element.