4 ms·
If the SBG issue really triggered the outage for the whole network I find it hard to believe that no one saw that problem beforehand. They probably thought that
by dx034 9y ago
If the SBG issue really triggered the outage for the whole network I find it hard to believe that no one saw that problem beforehand. They probably thought that this was too unlikely to happen or that there are other failovers but never tested them properly.
No expert on the field but that's the first time I can remember that a provider of that size loses connection to most of their data centres at once. That can happen with one product (eg S3 failure) but datacentre switches should work even if the rest is on fire.
- nkkollaw 9y agoWhat is SBG? A CDN of some sort?
- pfg 9y agoIt's the location of one of their data centers. SBG for Strasbourg.
- seszett 9y agoIt's a collection of datacenters in Strasbourg.
- nkkollaw 9y agoAh, gotcha. So, that one datacenter caused all other datacenters to die..?
- seszett 9y agoIt's supposed to be two separate incidents: power going down in Strasbourg, and fiber network equipment going down in Roubaix (the main center of OVH's network) due to a "software bug". It's explained here https://twitter.com/olesovhcom/status/928587258583748609 https://twitter.com/olesovhcom/status/928587258583748609 in French, they might post an English-language translation soon.
- nkkollaw 9y agoThanks!
- zaarn 9y agoIn this case, I wouldn't be to hard on them. As it appears they lost their main power line, the backup power line and both generators failed and one generator has been restarted now.
- dx034 9y agoSo what? Losing main power is a standard case for any DC. That's why you have generators. Even a generator failure is nothing out of the ordinary. But that no generators in a DC work kind of indicates that they don't test them as often as you would expect. They just announced that they want to be a "hypercloud" provider on the scale of AWS and Google Cloud. I really hope that a power failure in Virginia couldn't bring down all of AWS.
- zaarn 9y agoThe generators did work but they failed. Both of them. I can not imagine they weren't tested. But even the most rigorous testing can never reduce the total failure risk to 0. It seems OVH just got very very unlucky.
- jbb67 9y agoWhen I've seen things like this before, it's often been the switchover hardware that fails, not the generator as such. it's much harder to test that as you don't want to tell your customers "sorry your sever went down, we were just testing if the switchover worked and it didn't"
- vtsingaras 9y agoExactly that, we too suffered a power loss at our DC that was due to faults in the power supervising and switch-over circuits; we test both generators weekly.
- hinkley 9y agoI worked for a company doing mostly on-premises (wireless) carrier software years back but they still wanted our own server room to be 3 nines for reasons lost to me now. They had installed new electrical circuits high on the walls so that we could survive minor flooding incidents. So our lead Ops guy is unplugging half the redundant power supplies and plugging them into the new circuits, but a few critical servers are single PSU still, on UPS units. This is how he discovers that one of our PSUs has rotted and only has about five seconds of reserve power in it. Big outage, no bueno.
- pfg 9y ago> No expert on the field but that's the first time I can remember that a provider of that size loses connection to most of their data centres at once. It happened to GCE last year[1], though it only lasted 18 minutes. [1]: https://status.cloud.google.com/incident/compute/16007?post-mortem https://status.cloud.google.com/incident/compute/16007?post-...
- dx034 9y agoBut that's one product, not the whole DC. As I understand the post, other services worked correctly during that time.