13 ms·
OVH outage explained
- jedisct1 9y agoGreat summary. That's for the SBG outage, not the RBX one though.
- praseodym 9y agoDetails about the RBX outage were already posted earlier: http://travaux.ovh.net/?do=details&id=28244#comment35217 http://travaux.ovh.net/?do=details&id=28244#comment35217
- folbec 9y agoAccording to their maintenance postings, RBX was an upgrade problem to the software driver of the optical fiber : "RBX: We had a problem on the optical network that allows RBX to be connected with the interconnection points we have in Paris, Frankfurt, Amsterdam, London, Brussels. The origin of the problem is a software bug on the optical equipment, which caused the configuration to be lost and the connection to be cut from our site in RBX. We handed over the backup of the software configuration as soon as we diagnosed the source of the problem and the DC can be reached again. The incident on RBX is fixed. With the manufacturer, we are looking for the origin of the software bug and also looking to avoid this kind of critical incident. "
- deleted 9y ago[deleted]
- deleted 9y ago[deleted]
- tyingq 9y agoVery transparent, kudos for that. A reminder to everyone, though, that the brochure doesn't always match reality. OVH pitches itself as a global data center leader, with 20 facilities, 2 of them amongst the largest in the world. Not to harp on them, just to say that having some backup plan on a separate provider is always a good idea, no matter who your primary is.
- bringtheaction 9y agoAlso when you decide on a backup, make sure your backup is not hosting their servers in the same DCs as is your primary, or worse yet that your backup is renting their servers from the same people you rent your primary from.
- jeremyjh 9y agoYour backup site should be in a different region - e.g. if your primary is eastern US your backup should be central or western US. Otherwise you are vulnerable to various natural and man-made disasters.
- vidarh 9y agoMy lesson in this was not servers, but networks. Back in the mid 90's at some point, Europe largely lost network connectivity to the US for about a day (this is all from recollection; I ran an ISP at the time, and we were affected, but I'm sure I've gotten the occasional detail wrong). Not entirely - there was significant capacity that was technically operational, but in practical terms large subsets of European networks were cut off. The reason was problem with one of Sprint's cables. It was carrying a lot of Nordunet traffic. Nordunet is/was the Nordic joint university network infrastructure. At the time, for historical reasons, Nordunet had more network capacity than most of the rest of Europe combined (consider that Norway was the second international connection to Arpanet in the early 70's due to the seismic array at Kjeller outside Oslo that provided essential data on Soviet nuclear tests; this made money flow to network research in Norway, and a joint Nordic effort let to significant investment in the other Nordic countries too; couple it with early work in Sweden, where they ended up hosting D-Gix - for many years the largest internet exchange point outside the US). So Sprints cable went down, and everyone started re-routing. Problem #1: Everyone had high capacity to D-Gix or other exchange point in the Nordic countries. These connections were now flooded by traffic from the Nordic networks where people were used to being able to saturate their 100Mbps network connections (it was such a downer starting my ISP, and going from 100Mbps at university to 512kbps aggregate outbound capacity at our offices). Except most of the links connecting to D-Gix where 34Mbps or less, and there were not that many of them, and they were connecting entire universities or entire countries... So connectivity within Europe started slowing down. Problem #2: Most of the links from elsewhere in Europe to the US were 8Mbps or less, with maybe a couple ones faster than that, and there were not that many of them. Certainly not enough to compensate for several hundred Mbps of lost capacity. Problem #3: Many of those links got so saturated and slow that failsafes were tripped and traffic started getting routed to their expensive backup links (e.g. people paying for a port and then paying for burst a 95th percentile basis and the like with expectation of normally not using them). Except, as it turns out, many of said backup links had been purchased from Sprint. Over that cable. Today the number of cables and overall capacity is vastly higher, with many more providers, so I don't think the same could happen again, and D-Gix is no longer as important as it was (though it's still one of the largest international interchange points), but as we sat there pinging and trying to figure out what was up, it was a vivid lesson in ensuring your backups are actually sufficiently separate from your main systems that they stand a chance of working.
- duncanawoods 9y agoWhy are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?
- cjg_ 9y agoIn this case the generators themselves worked, just not the mechanism to cut over to them.
- duncanawoods 9y agoI include that bit of the system in "backup generator" to distinguish it from "generator".
- tyingq 9y agoGenerally called a "static switch", and they are prone to issues. Here's one: https://www.aegps.com/en/products/ups-accessories/static-transfer-switch/ https://www.aegps.com/en/products/ups-accessories/static-tra...
- dx034 9y agoAre they automatically a single point of failure or can you install several of those on one site?
- Xylakant 9y agoMost of the time, it's not the actual generator that fails. It's often either the part that detects power failure (fluctuations, brownouts, ... are harder to detect than total failure) or the switchover hardware to the generator. If you want to switch 20KV, you're really handling an awful lot of energy. Physics kinda get in the way there. Switching paket based networks is often much easier because you can just hold on to the packet and retransmit if a line fails, that's why it's much rarer to have total network failures.
- ngrilly 9y agoI have a hard time understanding why they needed humans babysitting the restart of servers and services. Servers and services are supposed to automatically restart after the power came back. This implies OVH doesn't regularly test server restarts with something like Netflix Chaos Monkey. Moreover, the power lines were not really redundant in SBG, and regarding the network downtime in RBX, it looks like the network core is not really redundant either. I'd be curious to know how competitors like AWS and Google Cloud mitigate this kind of problems.
- icebraining 9y agoOVH runs servers for their clients, do you really expect them to randomly restart their clients' servers? Netflix is not a hosting provider, the situations are not comparable.
- ngrilly 9y agoThe comment in the OVH post-mortem suggests it is about internal servers and services. You're right about customer bare-metal servers. They can't restart them randomly.
- wereHamster 9y agoI would be pissed if my hosting provider randomly restarted my servers, taking them offline for minutes each time.
- drdebug 9y agoSo would I, but that actually makes me realize that I should be doing more of what netflix is doing too and ensure I can randomly restart servers without affecting service.
- tete 9y agoIsn't choasmonkey just randomly restarting virtual servers on Amazon? Kudos to Netflix, but restarting a virtual server vs a physical or a whole data center are different things. I think every company that cares a bit about high availability knocks out stuff randomly or at least in different ways, does stuff like introducing packet loss, etc. It's another layer and another thing to test that on service/virtual server layer than on close to physical layers. Of course, one should test that too and it's nowhere near impossible, but Chaosmonkey is for a somewhat different use case. Also the "article" mentions that tests are done.
- tgtweak 9y agoAWS had an identical failure with their ATS(automatic transfer switch) system which took out an availability zone. After some forensics and a code review of the plc software, realized it was an omission in that system. They now make their own if I remember correctly. https://image.slidesharecdn.com/cpn208failuresatscale-121129215220-phpapp01/95/cpn208-failures-at-scale-how-to-ride-through-them-aws-re-invent-2012-19-638.jpg?cb=1434493281 https://image.slidesharecdn.com/cpn208failuresatscale-121129...
- kyledrake 9y agoMy datacenter has 2 power channels, coming from two different power grids. They have never had a full outage, even when a substation two blocks away started on fire. Even if we lost power on one power channel, we would still be operational. Even if the backup power failed. That's the reason you use redundant power. If your "cloud provider" is charging you a lot but not providing your server redundant power from two grids to redundant PSUs, I don't think you're getting the best available design. In this case, it looks like OVH just threw all the power onto a single channel, which allowed a failover bug to bring down the datacenter.
- Cyph0n 9y agoThe power grid in and of itself is a highly interconnected and redundant system. The fact that a substation fire didn't bring down your datacenter doesn't necessarily mean that the "redundant" setup was responsible. Unless the two grids are fully isolated (e.g., from two different countries), there is always going to be some overlap in the two power sources.
- kyledrake 9y agoThe trick to building good datacenters is you make as many things redundant as possible, and it reduces the chance of one thing taking the entire datacenter down. Being connected to two external power providers is not the reason our datacenter stayed up, it's one of the many carefully designed redundancies that all add up to higher availability. When you have an A/B isolated power datacenter, you can (and my datacenter regularly does) shut off each channel to test that the backup systems are working, without worrying that a failed test will bring down literally every customer in the datacenter. That testing would have likely caught this problem at OVH if they were able to do it, but they instead decided to just toss all the power into a single channel, making it impossible to do testing like this without a decent chance of widespread outages. So they just had to assume it would work, until it didn't. Because our power channels are isolated, there's almost no difference between turning each one off separately, and then turning both of them off at the same time. So if we were to lose external power on both (which would be a history books blackout for how this DC gets its power, including an almost direct connection to a hydro-electric dam), you would have two well-tested backup power channels ready to go. Nothing is ever a guarantee, but I like my chances.
- cesarb 9y agoI wonder how easy it is to really test these failover systems. Even if you disconnect the external power (which is risky, since you lose redundancy), it will be a "clean" disconnection, with the power going neatly to zero. I've found a few post-mortems of high-voltage faults, and the waveforms go crazy during the fault, which could lead to failure modes in the transfer switch which wouldn't be found on a "clean" disconnection.
- solatic 9y agoYou'd think that somebody'd start selling some kind of generator that you could hook up the systems to that would generate various forms of unclean power on purpose, precisely so that you could test those rare failure modes.
- gedrap 9y agoThat applies to other kinds of reliability testing too, especially everything that goes through a network. It's easy to handle when something is completely unavailable (i.e. instant error), but when something, be it a database or some endpoint, is available but horribly slow, that's a whole different thing.
- cperciva 9y agoSeems like we need systems designed with "suicide" mechanisms built in, so that if they detect that they have a poor quality of life (err, I mean, that they're providing a poor quality of service) they'll shut down completely.
- smoyer 9y agoThe circuit-breaker pattern is commonly used to make the affected system appear off-line to it's upstream dependents - you don't need to actually shut the server down completely and will probably want to log into it to see what's happening. If the dependent system can't function without the affected system, it should also indicate an error and further upstream systems should detect it. In your house, you have the circuit-breaker pattern implemented in hardware. And upstream of your house there are many more layers of circuit breakers that generally increase in size until they reach a point where there is redundancy. A circuit-breaker going off in your house protects the other circuits in your house, a circuit-breaker going off on your street protects the rest of your neighborhood. Industrial circuit-breakers are commonly a combination of hardware and software. Personally, I've lost more equipment due to brown-outs than any other cause. If you have equipment you don't want to experience a brown-out, program the breaker to cut off power completely.
- jijji 9y agoEvery small datacenter has a fake diesel generator that doesn't work. I worked for one datacenter and they purposely bought a non-working diesel generator and moved it into a room with the sole purpose to pitch it to investors that they had a working secondary power source.
- laumars 9y ago"Every"? I know for a fact that we don't because we've fallen back to the redundant generator before and it's worked flawlessly. (pity the same couldn't have been said for some of our other redundant systems).
- otterley 9y agoHow on earth did you go from "I saw this thing once" to "everyone does it"?
- shrike 9y agoWhen I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (short of danger to humans). Generators are an irrelevant cost compared to the cost of an outage. The manufacturer refused, even when we offered to buy them without warranties. I don't know if they still do but at that point AWS started to buy basically the same generators straight from China and writing their own firmware. There is also a great story about an AWS authored firmware update on a set of louver microcontrollers causing a partial outage. AWS really likes to own the entire stack. *It has been years since I left, details are a bit fuzzy but I think I've got these right.
- walshemj 9y agoAnd at AWS scale you can always have a human team present to provide a human back up the cost of having 2 or 3 people on an overnight shift is trivial compared to the down time costs.
- dx034 9y agoI wonder why they couldn't do that at OVH. The data centres are always manned and they had 8 minutes on batteries. Why didn't they detect after 20 seconds that generators don't start despite power failure and triggered a manual failover? Sounds like something the staff should be able to do?
- melq 9y agoDo you know why its harder for a generator to power an unloaded circuit? Just curious.
- deleted 9y ago[deleted]
- lima 9y ago
- rmdoss 9y agoI have been a big OVH advocate (cheap prices + free DDoS protection + good hardware) and have been using them for years. However, I am really frustrated by the lack of communication during this incident. All I had was the tweets from the founder, while their status page was down and I had no other place to reach out to them (phone + ticket system down). What is worse is that they treat the individual SBGs (SBG1, SBG2, etc) as individual availability zones, when they truly are not. I had an application using their vrack on different SBGs availability zones and they all went down. Anyway, glad they are fixing it, and glad for the clarification that they don't have real availability zones. Will design my applications better next time.
- therealmarv 9y agoCan I ask if vrack itself stayed online with non SBG instances inside? I'm just curious if this also affects their network wide services like vrack or public cloud instances outside SBG.
- tgtweak 9y agoIf power to network gear was down, then it probably affected vrack connectivity, likely on the entirety of sbg.
- bobby_the_whale 9y agoThey cut corners. If a company has done this once in the past, why would one ever trust them again? Building a reputation takes a long time and losing it just takes one power outage.
- patrickg_zill 9y agoDatacenter I was using for 5 years: once per month, employees on site, power is cut/disrupted somehow, UPS kicks in, within 60 seconds the 750kw diesel genset is supposed to be running and taking load. Zero problems in that 5 year period due to power...