5 ms·

In this case, I wouldn't be to hard on them. As it appears they lost their main power line, the backup power line and both generators failed and one generator h

by zaarn 9y ago

In this case, I wouldn't be to hard on them. As it appears they lost their main power line, the backup power line and both generators failed and one generator has been restarted now.

dx034 9y ago
So what? Losing main power is a standard case for any DC. That's why you have generators. Even a generator failure is nothing out of the ordinary. But that no generators in a DC work kind of indicates that they don't test them as often as you would expect. They just announced that they want to be a "hypercloud" provider on the scale of AWS and Google Cloud. I really hope that a power failure in Virginia couldn't bring down all of AWS.
- zaarn 9y ago
  The generators did work but they failed. Both of them. I can not imagine they weren't tested. But even the most rigorous testing can never reduce the total failure risk to 0. It seems OVH just got very very unlucky.
- jbb67 9y ago
  When I've seen things like this before, it's often been the switchover hardware that fails, not the generator as such. it's much harder to test that as you don't want to tell your customers "sorry your sever went down, we were just testing if the switchover worked and it didn't"
  - vtsingaras 9y ago
    Exactly that, we too suffered a power loss at our DC that was due to faults in the power supervising and switch-over circuits; we test both generators weekly.
  - hinkley 9y ago
    I worked for a company doing mostly on-premises (wireless) carrier software years back but they still wanted our own server room to be 3 nines for reasons lost to me now. They had installed new electrical circuits high on the walls so that we could survive minor flooding incidents. So our lead Ops guy is unplugging half the redundant power supplies and plugging them into the new circuits, but a few critical servers are single PSU still, on UPS units. This is how he discovers that one of our PSUs has rotted and only has about five seconds of reserve power in it. Big outage, no bueno.
- matthewmacleod 9y ago
  It doesn't matter how much testing you do – unfortunately some things can still fail. I'm much more interested in why the failure cascaded to other data centres – that's exactly what shouldn't happen.
  - hinkley 9y ago
    Didn’t AWS have a cascading failure last year because of a missing interlock on a script to take part of a data center offline? They discovered that it takes a lot longer than they remembered to reintroduce that many machines to the cluster. They came away from that incident with two or three action items.
hvidgaard 9y ago
What is the point of backup generators if you do not verify that they work every so often? I have a very hard time believing that they actually tested that they worked, because a failure of not one, but both of them.
- cjsuk 9y ago
  To be fair people do test generators on a monthly schedule usually. Problem you find is it’s getting colder now so any problems are amplified suddenly. Might have been entirely tested a couple of weeks ago.
  - hvidgaard 9y ago
    If your generators are not reliable in the cold, the issue is the placement of them. This is something basic to account for.
    - cjsuk 9y ago
      Yes and no. What the generator vendor says when they sell you the generator and what actually happens 5 years down the line are two different things.
    - zaarn 9y ago
      The problem is usually not "not reliable in cold" but rather, the generators are X years old and the temperature is now changing in Europe from "mostly warm" to "warm over the day and icecold in the night" and finally aiming for "icecold all day", which means any equipment exposed will go through rather severe temperature changes. While generators are usually able to handle this with sufficiently low failure risk, the risk is increase due to the changing temperature
      - dx034 9y ago
        It wasn't even freezing overnight at Strasbourg [1], I doubt that the cold could cause any effects at 5C. Maybe they were run once a month but never tested under load? 2x20kv lines failing will have put them at near maximum load immediately, perhaps they weren't designed for that. [1] https://www.accuweather.com/en/fr/strasbourg/131836/daily-weather-forecast/131836?day=1 https://www.accuweather.com/en/fr/strasbourg/131836/daily-we...

by zaarn 9y ago

In this case, I wouldn't be to hard on them. As it appears they lost their main power line, the backup power line and both generators failed and one generator has been restarted now.

dx034 9y ago

So what? Losing main power is a standard case for any DC. That's why you have generators. Even a generator failure is nothing out of the ordinary. But that no generators in a DC work kind of indicates that they don't test them as often as you would expect. They just announced that they want to be a "hypercloud" provider on the scale of AWS and Google Cloud. I really hope that a power failure in Virginia couldn't bring down all of AWS.

zaarn 9y ago
The generators did work but they failed. Both of them. I can not imagine they weren't tested. But even the most rigorous testing can never reduce the total failure risk to 0. It seems OVH just got very very unlucky.
jbb67 9y ago
When I've seen things like this before, it's often been the switchover hardware that fails, not the generator as such. it's much harder to test that as you don't want to tell your customers "sorry your sever went down, we were just testing if the switchover worked and it didn't"
- vtsingaras 9y ago
  Exactly that, we too suffered a power loss at our DC that was due to faults in the power supervising and switch-over circuits; we test both generators weekly.
- hinkley 9y ago
  I worked for a company doing mostly on-premises (wireless) carrier software years back but they still wanted our own server room to be 3 nines for reasons lost to me now. They had installed new electrical circuits high on the walls so that we could survive minor flooding incidents. So our lead Ops guy is unplugging half the redundant power supplies and plugging them into the new circuits, but a few critical servers are single PSU still, on UPS units. This is how he discovers that one of our PSUs has rotted and only has about five seconds of reserve power in it. Big outage, no bueno.
matthewmacleod 9y ago
It doesn't matter how much testing you do – unfortunately some things can still fail. I'm much more interested in why the failure cascaded to other data centres – that's exactly what shouldn't happen.
- hinkley 9y ago
  Didn’t AWS have a cascading failure last year because of a missing interlock on a script to take part of a data center offline? They discovered that it takes a lot longer than they remembered to reintroduce that many machines to the cluster. They came away from that incident with two or three action items.

hvidgaard 9y ago

What is the point of backup generators if you do not verify that they work every so often? I have a very hard time believing that they actually tested that they worked, because a failure of not one, but both of them.

cjsuk 9y ago
To be fair people do test generators on a monthly schedule usually. Problem you find is it’s getting colder now so any problems are amplified suddenly. Might have been entirely tested a couple of weeks ago.
- hvidgaard 9y ago
  If your generators are not reliable in the cold, the issue is the placement of them. This is something basic to account for.
  - cjsuk 9y ago
    Yes and no. What the generator vendor says when they sell you the generator and what actually happens 5 years down the line are two different things.
  - zaarn 9y ago
    The problem is usually not "not reliable in cold" but rather, the generators are X years old and the temperature is now changing in Europe from "mostly warm" to "warm over the day and icecold in the night" and finally aiming for "icecold all day", which means any equipment exposed will go through rather severe temperature changes. While generators are usually able to handle this with sufficiently low failure risk, the risk is increase due to the changing temperature
    - dx034 9y ago
      It wasn't even freezing overnight at Strasbourg [1], I doubt that the cold could cause any effects at 5C. Maybe they were run once a month but never tested under load? 2x20kv lines failing will have put them at near maximum load immediately, perhaps they weren't designed for that. [1] https://www.accuweather.com/en/fr/strasbourg/131836/daily-weather-forecast/131836?day=1 https://www.accuweather.com/en/fr/strasbourg/131836/daily-we...