3 ms·

I used to work for iServer (later bought by Verio; in full-disclosure, I'm now a contractor for Verio). I remember one time the city had shifted their output by

by scottw 17y ago

I used to work for iServer (later bought by Verio; in full-disclosure, I'm now a contractor for Verio). I remember one time the city had shifted their output by 90° (or something like that...this was years ago). Our datacenter UPS was tied right into the 3-phase output and didn't recognize that the power was still flowing from the city. The diesel engines fired up and started powering the datacenter (redundantly). When the gas ran out a few hours later, everything started shutting down. It was terrible. Hundreds of servers with 600-800+ days uptime all went down and stayed down for hours. We were shuttling people from the office down to the datacenter to fsck and bring machines back up. It was a long night and long next week issuing credits to customers who had lost data. Today I've got servers with Verio, Bluehost, and LayeredTech. They've all been great for the most part, but none of them have been near perfect. I know that most hosting companies I've worked for or dealt with put uptime near the top of their list of priorities because they know it's an easily measured number people bank on. I'm sure RS will get their act together and be as solid as they've typically been in the past.

MoeDrippins 17y ago
Genuine question here. I've heard these stories before (and lived through one at a company, but under very different circumstances), and it always amazes me that anyone would let a generator run out of diesel. How can that happen? I understand the fuel tanks on these things are enormous and you can't just run to QT and get a couple gallons to tide you over, but they had to have been filled by SOMETHING initially, right? How can it be that whatever that was, can't be repeated? Do HA plans not cover this contingency? Do they not specify that someone immediately calls a fuel truck the SECOND the generator kicks on?
- scottw 17y ago
  Well, in this case because the power never went out (the UPS only thought it did), none of our usual "the freakin' power is out—send out the diesel trucks and keep the tank full" alarms went off. It was truly an unexpected case (which was quickly fixed in the UPS hardware), but you know, life is one unexpected case after another, often coming in waves and in bizarre combinations. I don't believe in 100% uptime anymore :)
  - moe 17y ago
    It was truly an unexpected case (which was quickly fixed in the UPS hardware), but you know, life is one unexpected case after another The word you are looking for is incompetence. An UPS, even a datacenter scale UPS, is not exactly rocket surgery. If your story is true and the device failed over to diesel without notifying anyone then that's not only an epic engineering failure but also an epic fiscal failure for whoever is liable (perhaps the UPS vendor). Damages from a full DC blackout easily run into the hundred thousands of dollars per hour, not even counting the unbillable shockwave of "our website is down" multiplied by hundreds of customers. It's a ridiculously expensive "Oops" that easily dwarfs the cost for deploying a proper UPS with proper testing and proper procedures in first place. Btw the CAT in our Level3 datacenter over here has a big horn and a flashlight on the side. My naive self wants to believe they are there for situations such as the diesel running out...
    - scottw 17y ago
      It was a UPS failure, yes, but this was in the late 90s when the UPS industry wasn't as robust as it is now. Competency is what you get when you learn from your incompetency. We learned something valuable from it (and our UPS people did too). I also learned to cut people (and some companies) a little slack when I've been down the road they're on. We had a lot of customers who didn't cut us any slack, took their refund and left, which was their prerogative to do so, but found out for themselves that no company has 100% uptime (many came back, lucky for us).
  - deleted 17y ago
    [deleted]

by scottw 17y ago

MoeDrippins 17y ago

Genuine question here. I've heard these stories before (and lived through one at a company, but under very different circumstances), and it always amazes me that anyone would let a generator run out of diesel. How can that happen? I understand the fuel tanks on these things are enormous and you can't just run to QT and get a couple gallons to tide you over, but they had to have been filled by SOMETHING initially, right? How can it be that whatever that was, can't be repeated? Do HA plans not cover this contingency? Do they not specify that someone immediately calls a fuel truck the SECOND the generator kicks on?

scottw 17y ago
Well, in this case because the power never went out (the UPS only thought it did), none of our usual "the freakin' power is out—send out the diesel trucks and keep the tank full" alarms went off. It was truly an unexpected case (which was quickly fixed in the UPS hardware), but you know, life is one unexpected case after another, often coming in waves and in bizarre combinations. I don't believe in 100% uptime anymore :)
- moe 17y ago
  It was truly an unexpected case (which was quickly fixed in the UPS hardware), but you know, life is one unexpected case after another The word you are looking for is incompetence. An UPS, even a datacenter scale UPS, is not exactly rocket surgery. If your story is true and the device failed over to diesel without notifying anyone then that's not only an epic engineering failure but also an epic fiscal failure for whoever is liable (perhaps the UPS vendor). Damages from a full DC blackout easily run into the hundred thousands of dollars per hour, not even counting the unbillable shockwave of "our website is down" multiplied by hundreds of customers. It's a ridiculously expensive "Oops" that easily dwarfs the cost for deploying a proper UPS with proper testing and proper procedures in first place. Btw the CAT in our Level3 datacenter over here has a big horn and a flashlight on the side. My naive self wants to believe they are there for situations such as the diesel running out...
  - scottw 17y ago
    It was a UPS failure, yes, but this was in the late 90s when the UPS industry wasn't as robust as it is now. Competency is what you get when you learn from your incompetency. We learned something valuable from it (and our UPS people did too). I also learned to cut people (and some companies) a little slack when I've been down the road they're on. We had a lot of customers who didn't cut us any slack, took their refund and left, which was their prerogative to do so, but found out for themselves that no company has 100% uptime (many came back, lucky for us).
- deleted 17y ago
  [deleted]