3 ms·

Explosion At ThePlanet Datacenter Drops 9,000 Servers, 7,500 Customers

32 points by nickb 18y ago

dmix 18y ago
Thats interesting, just yesterday I was reading into datacenters and how they are engineered. The fire prevention techniques are usually really advanced and that may be why none of the servers were damaged.
- metachor 18y ago
  Indeed. Many data-centers have a multi-tiered approach to fire-suppression. This usually includes a chemical suppressant as first line (often a de-oxygenate that suppresses fires by removing oxygen from the room and does not damage servers) as well as a preaction dry-pipe system as a backup (in this case the fire has gone too far to save the severs and the goal is to protect the rest of the building).
metachor 18y ago
This is an interesting lesson in disaster recovery planning for businesses using third-party datacenters for hosting. One common scenario is for a business with its own datacenter to use a private third-party datacenter (like the one in the article) as their off-site DR location. But what do you do when your DR location suffers an outage? Many companies will host in multiple datacenters to prevent outages in any one physical location from bringing down all of their operations. I suppose ultimate security could only be achieved using a distributed approach, but that may cost beyond what many businesses can afford (unless you are Google or Microsoft, for example).
- SwellJoe 18y ago
  Theoretically Google or Amazon could provide the same level of reliability to their hosted customers. But, in reality, uptime of servers at Amazon has been significantly lower than The Planet, even accounting for this outage. We have a couple of servers with The Planet, but luckily, they are in Dallas rather than Houston. This event did make me rethink our backup procedures, however. Backing up to another server in the same data center provides no protection against this kind of event, and though it sounds like there is no data loss, I'm sure a lot of folks were experiencing some pangs of regret yesterday before the full scope of the damage was understood.
  - metachor 18y ago
    I think a minimal necessity for a DR plan is to have at least three geographically seperated hosting locations with some sort of automatic replication or fail-over. Again, this costs a lot of money. I don't know whether the ec2/cloud-computing is comparable as of today.
bprater 18y ago
Bring on the cloud. Enough of this hosting stuff on a "single" machine craziness. Folks in 20 years are going to scratch their heads when we try to explain what we did back in the old days.
- delano 18y ago
  I think it'll be more like, "Wowzerz, hosting must have been hella stressful in those days, hey Pops".
- neilc 18y ago
  Bring on the cloud. A massive service outage is a reason to encourage the move to cloud computing? Such an architecture would only be more vulnerable to service failures outside the user's control, not less.
  - johnrob 18y ago
    Not if your virtual instances can be replicated to another data center (an EC2 instance is location agnostic).
tlrobinson 18y ago
The headline makes it sound worse than it was... no servers were damaged in the explosion, power was just knocked out.
- reggplant 18y ago
  Well it was powerful enough to knock down 3 walls surrounding the power room!
- jonknee 18y ago
  Perhaps the servers were not near the explosion?
- metachor 18y ago
  Having the power knocked out for several days IS that bad. Without power the servers are offline and no long running whatever services they were meant to be providing. This means lost money for whoever those services belonged to (and lost money for those people's clients, potential lawsuits, etc). Preventing this sort of downtime is precisely why many businesses will host in a professional datacenter instead of "in house".
herdrick 18y ago
"They estimate they will be up and running by Sunday afternoon." Still down.
herdrick 18y ago
More comments here: http://news.ycombinator.com/item?id=205709 http://news.ycombinator.com/item?id=205709

32 points by nickb 18y ago

dmix 18y ago

Thats interesting, just yesterday I was reading into datacenters and how they are engineered. The fire prevention techniques are usually really advanced and that may be why none of the servers were damaged.

metachor 18y ago
Indeed. Many data-centers have a multi-tiered approach to fire-suppression. This usually includes a chemical suppressant as first line (often a de-oxygenate that suppresses fires by removing oxygen from the room and does not damage servers) as well as a preaction dry-pipe system as a backup (in this case the fire has gone too far to save the severs and the goal is to protect the rest of the building).

metachor 18y ago

This is an interesting lesson in disaster recovery planning for businesses using third-party datacenters for hosting. One common scenario is for a business with its own datacenter to use a private third-party datacenter (like the one in the article) as their off-site DR location. But what do you do when your DR location suffers an outage? Many companies will host in multiple datacenters to prevent outages in any one physical location from bringing down all of their operations. I suppose ultimate security could only be achieved using a distributed approach, but that may cost beyond what many businesses can afford (unless you are Google or Microsoft, for example).

SwellJoe 18y ago
Theoretically Google or Amazon could provide the same level of reliability to their hosted customers. But, in reality, uptime of servers at Amazon has been significantly lower than The Planet, even accounting for this outage. We have a couple of servers with The Planet, but luckily, they are in Dallas rather than Houston. This event did make me rethink our backup procedures, however. Backing up to another server in the same data center provides no protection against this kind of event, and though it sounds like there is no data loss, I'm sure a lot of folks were experiencing some pangs of regret yesterday before the full scope of the damage was understood.
- metachor 18y ago
  I think a minimal necessity for a DR plan is to have at least three geographically seperated hosting locations with some sort of automatic replication or fail-over. Again, this costs a lot of money. I don't know whether the ec2/cloud-computing is comparable as of today.

bprater 18y ago

Bring on the cloud. Enough of this hosting stuff on a "single" machine craziness. Folks in 20 years are going to scratch their heads when we try to explain what we did back in the old days.

delano 18y ago
I think it'll be more like, "Wowzerz, hosting must have been hella stressful in those days, hey Pops".
neilc 18y ago
Bring on the cloud. A massive service outage is a reason to encourage the move to cloud computing? Such an architecture would only be more vulnerable to service failures outside the user's control, not less.
- johnrob 18y ago
  Not if your virtual instances can be replicated to another data center (an EC2 instance is location agnostic).

tlrobinson 18y ago

The headline makes it sound worse than it was... no servers were damaged in the explosion, power was just knocked out.

reggplant 18y ago
Well it was powerful enough to knock down 3 walls surrounding the power room!
jonknee 18y ago
Perhaps the servers were not near the explosion?
metachor 18y ago
Having the power knocked out for several days IS that bad. Without power the servers are offline and no long running whatever services they were meant to be providing. This means lost money for whoever those services belonged to (and lost money for those people's clients, potential lawsuits, etc). Preventing this sort of downtime is precisely why many businesses will host in a professional datacenter instead of "in house".

herdrick 18y ago

"They estimate they will be up and running by Sunday afternoon." Still down.

herdrick 18y ago

More comments here: http://news.ycombinator.com/item?id=205709 http://news.ycombinator.com/item?id=205709