5 ms·

Yeah the rackspace outage freaked us out a little. Seems like it would be nice if they could notify the customers before the customers discovered the downtime

by catch23 17y ago

Yeah the rackspace outage freaked us out a little. Seems like it would be nice if they could notify the customers before the customers discovered the downtime themselves. If Rackspace emailed their customers at the first sign of downtime, customers would probably be more patient.

idlewords 17y ago
Without knowing the details of the incident, it's hard to blame them. Personally I prefer the first responders to focus all their attention on diagnosing the problem. Once other people arrive on the scene (which in a hosting company I imagine is not many minutes after pagers start going off) they can take charge of notification and posting updates.
- gommm 17y ago
  Yes but in that case they took at least 40 minutes before notifying customers and posting on their status blog... I had to contact them on the live chat to be able to know what was happening... And they could have a script that directly posts to the status page something like "A lot of our servers are down, we are working on it please be patient" when they detect some problem wouldn't take long...
- catch23 17y ago
  Usually the best way to do customer service is to first admit the problem, then solve it. I'm sure not everyone who works at Rackspace is technical, so someone could initiate the email notification 5 minutes into the downtime, maybe that could be done via nagios on an external system even. If the system sees that 50% of the servers are down, notify all customers, regardless if you were directly affected or not.
human_v2 17y ago
I work for a web host (not this one). From my experience, this is generally what is going on during downtime (note: times are BEST CASE scenario): T0: Servers go down T5-15: Someone realizes something is broken. T15-45: Initial someone calls another someone to tell them it's broken because the night guy probably can't fix it all by himself. T60-70: All the engineers are woken up and know OF the problem. T90: Engineers figure out WHAT the problem is. T120: Engineers fix problems. T150: More problems arise. T180: All problems are solved, no more customers are mad. Moral of the Story: As you can see, it already is a very involved process to get servers running again. Now, as a customer, you want an email to notify you that services are down. First, the admin needs to find out exactly which customers are affected and then write up a nice happy email to you letting you know we're working on it. Do you want a happy email or do you want your servers fixed? Your call. Downtime is chaos. Do not ask to be notified of chaos.
- theli0nheart 17y ago
  Honestly, I don't need well-drafted prose or a happy email to alert me of a server outage. I could care less. They should have a pre-drafted document that let's them fill in blanks based on the problem at hand. I was one of these affected customers and can say that not being notified of the servers being down cost me a few hours of uptime. I would say that customer communication during chaos is nearly as important as fixing the problem itself. Just my two cents.
- shpxnvz 17y ago
  Do you want a happy email or do you want your servers fixed? Your call. It's not like the two are mutually exclusive. Are you really suggesting that the entire 180 minutes of effort by multiple engineers would have to be directed at writing a "happy email" rather than fixing the problem? The real reason, I imagine, is that if they automatically sent out an email to everyone who could be affected then more people would know of the problem. It's probably smart to accept a handful of customers being unhappy about being in the dark in return for the outage being exposed to far fewer customers.

by catch23 17y ago

idlewords 17y ago

Without knowing the details of the incident, it's hard to blame them. Personally I prefer the first responders to focus all their attention on diagnosing the problem. Once other people arrive on the scene (which in a hosting company I imagine is not many minutes after pagers start going off) they can take charge of notification and posting updates.

gommm 17y ago
Yes but in that case they took at least 40 minutes before notifying customers and posting on their status blog... I had to contact them on the live chat to be able to know what was happening... And they could have a script that directly posts to the status page something like "A lot of our servers are down, we are working on it please be patient" when they detect some problem wouldn't take long...
catch23 17y ago
Usually the best way to do customer service is to first admit the problem, then solve it. I'm sure not everyone who works at Rackspace is technical, so someone could initiate the email notification 5 minutes into the downtime, maybe that could be done via nagios on an external system even. If the system sees that 50% of the servers are down, notify all customers, regardless if you were directly affected or not.

human_v2 17y ago

I work for a web host (not this one). From my experience, this is generally what is going on during downtime (note: times are BEST CASE scenario): T0: Servers go down T5-15: Someone realizes something is broken. T15-45: Initial someone calls another someone to tell them it's broken because the night guy probably can't fix it all by himself. T60-70: All the engineers are woken up and know OF the problem. T90: Engineers figure out WHAT the problem is. T120: Engineers fix problems. T150: More problems arise. T180: All problems are solved, no more customers are mad. Moral of the Story: As you can see, it already is a very involved process to get servers running again. Now, as a customer, you want an email to notify you that services are down. First, the admin needs to find out exactly which customers are affected and then write up a nice happy email to you letting you know we're working on it. Do you want a happy email or do you want your servers fixed? Your call. Downtime is chaos. Do not ask to be notified of chaos.

theli0nheart 17y ago
Honestly, I don't need well-drafted prose or a happy email to alert me of a server outage. I could care less. They should have a pre-drafted document that let's them fill in blanks based on the problem at hand. I was one of these affected customers and can say that not being notified of the servers being down cost me a few hours of uptime. I would say that customer communication during chaos is nearly as important as fixing the problem itself. Just my two cents.
shpxnvz 17y ago
Do you want a happy email or do you want your servers fixed? Your call. It's not like the two are mutually exclusive. Are you really suggesting that the entire 180 minutes of effort by multiple engineers would have to be directed at writing a "happy email" rather than fixing the problem? The real reason, I imagine, is that if they automatically sent out an email to everyone who could be affected then more people would know of the problem. It's probably smart to accept a handful of customers being unhappy about being in the dark in return for the outage being exposed to far fewer customers.