3 ms·

The Planet faces 24 hours of downtime for 9000 servers in their primary datacenter

12 points by e1ven 18y ago

herdrick 18y ago
"Dear Valued Customers: This evening at 4:55 in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room Thankfully, no one was injured. In addition, no customer servers were damaged or lost. We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department. This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock. We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well. There is no impact in any of our other five data centers. I am sorry that this accident has occurred and apologize for the impact. Sincerely, Douglas J. Erwin Chairman & Chief Executive Officer" I am not happy.
- reggplant 18y ago
  Can't be helped really as they can't initiate their backup systems because of the fire departments orders. I see webfaction is down as a result :/
deleted 18y ago
[deleted]
Tichy 18y ago
"We were not allowed to activate our backup generator plan based on instructions from the fire department." That is the interesting bit. So I guess two datacenters in different physical locations are a necessity as a backup plan.
jncraton 18y ago
I feel bad for these admins. It's not really their fault, but I'm sure a lot of end users are going to give them a lot of crap.
brk 18y ago
EVERY datacenter eventually suffers outages. Either through acts of stupidity or nature. Having all of your eggs in one basket, even if that basket is labelled "Google" or "Amazon", WILL lead to issues like this. Also, "leasing" servers will also lead to headaches eventually. Leased servers, EC2, etc. are all fine while you're in a growth phase, but at some point if you want to run a big web enterprise, you have to DIY.
herdrick 18y ago
Over 24 hours now.
- ojbyrne 18y ago
  I've got servers up there now.
  - ojbyrne 18y ago
    though apparently not with reliable dns yet.
    - ojbyrne 18y ago
      To save others some time they put up info on alternate resolvers: NTT x.ns.verio.net 129.250.35.250 NTT y.ns.verio.net 129.250.35.251 H2 67.15.31.131 H2 66.98.240.131 http://forums.theplanet.com/index.php?s=&showtopic=90185&view=findpost&p=593360 http://forums.theplanet.com/index.php?s=&showtopic=90185...

12 points by e1ven 18y ago

"Dear Valued Customers: This evening at 4:55 in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room Thankfully, no one was injured. In addition, no customer servers were damaged or lost. We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department. This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock. We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well. There is no impact in any of our other five data centers. I am sorry that this accident has occurred and apologize for the impact. Sincerely, Douglas J. Erwin Chairman & Chief Executive Officer" I am not happy.

reggplant 18y ago
Can't be helped really as they can't initiate their backup systems because of the fire departments orders. I see webfaction is down as a result :/

deleted 18y ago

[deleted]

Tichy 18y ago

"We were not allowed to activate our backup generator plan based on instructions from the fire department." That is the interesting bit. So I guess two datacenters in different physical locations are a necessity as a backup plan.

jncraton 18y ago

I feel bad for these admins. It's not really their fault, but I'm sure a lot of end users are going to give them a lot of crap.

brk 18y ago

EVERY datacenter eventually suffers outages. Either through acts of stupidity or nature. Having all of your eggs in one basket, even if that basket is labelled "Google" or "Amazon", WILL lead to issues like this. Also, "leasing" servers will also lead to headaches eventually. Leased servers, EC2, etc. are all fine while you're in a growth phase, but at some point if you want to run a big web enterprise, you have to DIY.

herdrick 18y ago

Over 24 hours now.

ojbyrne 18y ago
I've got servers up there now.
- ojbyrne 18y ago
  though apparently not with reliable dns yet.
  - ojbyrne 18y ago
    To save others some time they put up info on alternate resolvers: NTT x.ns.verio.net 129.250.35.250 NTT y.ns.verio.net 129.250.35.251 H2 67.15.31.131 H2 66.98.240.131 http://forums.theplanet.com/index.php?s=&showtopic=90185&view=findpost&p=593360 http://forums.theplanet.com/index.php?s=&showtopic=90185...