6 ms·

Widespread Rackspace Outage. DFW datacenter unreachable.

6 points by pmikal 17y ago

If your server is on Rackspace, chances are it's unreachable. Support and staus lines down too. Major fail.

noodle 17y ago
slicehost is still up. edited for clarification: slicehost is owned by rackspace. thought it would be worthwhile to mention since a lot of people here use it; i know i double-checked just to make sure.
- ptomato 17y ago
  Slicehost in general may be up, but slices hosted in the DFW datacenter were down.
  - noodle 17y ago
    were there any?
    - ptomato 17y ago
      Yeah, it's an option now and (most?) new customers would be there by default now I believe.
pmikal 17y ago
Just got call from my rep. DFW datacenter is completely out. Utility company cut power and backup generators have failed. Supposedly they've been having problems with these generators for awhile now. Ugh, shouldn't be so hard to keep servers online....
- evgen 17y ago
  In addition to the power problems on individual servers that I hear people complain about this is their second major power-related failure for that site in less than two years. These guys expect us to trust them as a cloud computing vendor? Seriously?
  - uptown 17y ago
    Unless you're hosting everything yourself, it seems like you'd have the same problem whether it's using the cloud or not ... wouldn't you?
- TallGuyShort 17y ago
  Just curious if anyone knows - why'd the utility company cut their power? And is that the same reason for their previous power-related outage?
vaksel 17y ago
Just goes to show you that the 100% uptime guarantee is nothing but b.s. At least the other hosts are honest enough to offer 99.9999% Edit: Actually just checked, and apparently my host switched from the 99.9999% to 100% as well.
- scottw 17y ago
  It is b.s., but the good news is that you can often get a credit annually for whatever percentage they're down from their SLA (I don't know if RS does this). Before you buy from a web host, check their SLA and ask how they cover SLA breaches. Don't do business with a company that won't put their money where their claims are.
  - vaksel 17y ago
    Rackspace policy is obviously better since they charge a lot more. My host has: If Liquid Web is or is not directly responsible for causing the downtime, the customer will receive a credit for 10 times ( 1,000% ) the actual amount of downtime. This means that if your server is unreachable for 1 hour (beyond the 0.0% allowed), you will receive 10 hours of credit. Rackspace has: Rackspace Guaranty: We will credit your account 5% of the monthly fee for each 30 minutes of network downtime, up to 100% of your monthly fee for the affected server.
    - scottw 17y ago
      I foresee a sharp drop in July profits... unless they weasel out of it by differentiating between "uptime" (i.e., does the CPU have power) and "reachability" (i.e., "Our network died! Sorry about that, but no soup for you").
scottw 17y ago
I used to work for iServer (later bought by Verio; in full-disclosure, I'm now a contractor for Verio). I remember one time the city had shifted their output by 90° (or something like that...this was years ago). Our datacenter UPS was tied right into the 3-phase output and didn't recognize that the power was still flowing from the city. The diesel engines fired up and started powering the datacenter (redundantly). When the gas ran out a few hours later, everything started shutting down. It was terrible. Hundreds of servers with 600-800+ days uptime all went down and stayed down for hours. We were shuttling people from the office down to the datacenter to fsck and bring machines back up. It was a long night and long next week issuing credits to customers who had lost data. Today I've got servers with Verio, Bluehost, and LayeredTech. They've all been great for the most part, but none of them have been near perfect. I know that most hosting companies I've worked for or dealt with put uptime near the top of their list of priorities because they know it's an easily measured number people bank on. I'm sure RS will get their act together and be as solid as they've typically been in the past.
- MoeDrippins 17y ago
  Genuine question here. I've heard these stories before (and lived through one at a company, but under very different circumstances), and it always amazes me that anyone would let a generator run out of diesel. How can that happen? I understand the fuel tanks on these things are enormous and you can't just run to QT and get a couple gallons to tide you over, but they had to have been filled by SOMETHING initially, right? How can it be that whatever that was, can't be repeated? Do HA plans not cover this contingency? Do they not specify that someone immediately calls a fuel truck the SECOND the generator kicks on?
  - scottw 17y ago
    Well, in this case because the power never went out (the UPS only thought it did), none of our usual "the freakin' power is out—send out the diesel trucks and keep the tank full" alarms went off. It was truly an unexpected case (which was quickly fixed in the UPS hardware), but you know, life is one unexpected case after another, often coming in waves and in bizarre combinations. I don't believe in 100% uptime anymore :)
testkat 17y ago
HAHAHA -- another outage! Great electrical Engineering team, Lanham...

6 points by pmikal 17y ago

If your server is on Rackspace, chances are it's unreachable. Support and staus lines down too. Major fail.

slicehost is still up. edited for clarification: slicehost is owned by rackspace. thought it would be worthwhile to mention since a lot of people here use it; i know i double-checked just to make sure.

ptomato 17y ago
Slicehost in general may be up, but slices hosted in the DFW datacenter were down.
- noodle 17y ago
  were there any?
  - ptomato 17y ago
    Yeah, it's an option now and (most?) new customers would be there by default now I believe.

pmikal 17y ago

Just got call from my rep. DFW datacenter is completely out. Utility company cut power and backup generators have failed. Supposedly they've been having problems with these generators for awhile now. Ugh, shouldn't be so hard to keep servers online....

evgen 17y ago
In addition to the power problems on individual servers that I hear people complain about this is their second major power-related failure for that site in less than two years. These guys expect us to trust them as a cloud computing vendor? Seriously?
- uptown 17y ago
  Unless you're hosting everything yourself, it seems like you'd have the same problem whether it's using the cloud or not ... wouldn't you?
TallGuyShort 17y ago
Just curious if anyone knows - why'd the utility company cut their power? And is that the same reason for their previous power-related outage?

vaksel 17y ago

Just goes to show you that the 100% uptime guarantee is nothing but b.s. At least the other hosts are honest enough to offer 99.9999% Edit: Actually just checked, and apparently my host switched from the 99.9999% to 100% as well.

scottw 17y ago
It is b.s., but the good news is that you can often get a credit annually for whatever percentage they're down from their SLA (I don't know if RS does this). Before you buy from a web host, check their SLA and ask how they cover SLA breaches. Don't do business with a company that won't put their money where their claims are.
- vaksel 17y ago
  Rackspace policy is obviously better since they charge a lot more. My host has: If Liquid Web is or is not directly responsible for causing the downtime, the customer will receive a credit for 10 times ( 1,000% ) the actual amount of downtime. This means that if your server is unreachable for 1 hour (beyond the 0.0% allowed), you will receive 10 hours of credit. Rackspace has: Rackspace Guaranty: We will credit your account 5% of the monthly fee for each 30 minutes of network downtime, up to 100% of your monthly fee for the affected server.
  - scottw 17y ago
    I foresee a sharp drop in July profits... unless they weasel out of it by differentiating between "uptime" (i.e., does the CPU have power) and "reachability" (i.e., "Our network died! Sorry about that, but no soup for you").

scottw 17y ago

I used to work for iServer (later bought by Verio; in full-disclosure, I'm now a contractor for Verio). I remember one time the city had shifted their output by 90° (or something like that...this was years ago). Our datacenter UPS was tied right into the 3-phase output and didn't recognize that the power was still flowing from the city. The diesel engines fired up and started powering the datacenter (redundantly). When the gas ran out a few hours later, everything started shutting down. It was terrible. Hundreds of servers with 600-800+ days uptime all went down and stayed down for hours. We were shuttling people from the office down to the datacenter to fsck and bring machines back up. It was a long night and long next week issuing credits to customers who had lost data. Today I've got servers with Verio, Bluehost, and LayeredTech. They've all been great for the most part, but none of them have been near perfect. I know that most hosting companies I've worked for or dealt with put uptime near the top of their list of priorities because they know it's an easily measured number people bank on. I'm sure RS will get their act together and be as solid as they've typically been in the past.

MoeDrippins 17y ago
Genuine question here. I've heard these stories before (and lived through one at a company, but under very different circumstances), and it always amazes me that anyone would let a generator run out of diesel. How can that happen? I understand the fuel tanks on these things are enormous and you can't just run to QT and get a couple gallons to tide you over, but they had to have been filled by SOMETHING initially, right? How can it be that whatever that was, can't be repeated? Do HA plans not cover this contingency? Do they not specify that someone immediately calls a fuel truck the SECOND the generator kicks on?
- scottw 17y ago
  Well, in this case because the power never went out (the UPS only thought it did), none of our usual "the freakin' power is out—send out the diesel trucks and keep the tank full" alarms went off. It was truly an unexpected case (which was quickly fixed in the UPS hardware), but you know, life is one unexpected case after another, often coming in waves and in bizarre combinations. I don't believe in 100% uptime anymore :)

testkat 17y ago

HAHAHA -- another outage! Great electrical Engineering team, Lanham...