6 ms·
All servers at OVH North American datacenters offline due to fiber cut
- sithadmin 11y agoMultihome your edge, folks.
- api 11y agoGlad to see this instead of bashing on OVH. This can happen anywhere. Every data center is at the end of one or maybe a few very thin little cables that can be cut very easily.
- deleted 11y ago[deleted]
- joepie91_ 11y agoI don't really understand this reply. 1. What data are you comparing? AWS does not make network statistics public, and has had major entire-region outages in the past as well. 2. OVH is one of the biggest hosting companies in the world. What's the point of the "huge" adjective for AWS? 3. This really doesn't have anything to do with prices, per se. OVH is cheap because they automate everything heavily, not necessarily because their actual quality of service (aside from customer support) is worse. We don't have insight into the financials of either company, and such financials are typically very complex, so let's not make this into some pseudo-"you get what you pay for" argument.
- rstupek 11y agoI think he's referring to AWS which sites their datacenters and have multiple fiber connections situated such that cutting one doesn't bring down the entire datacenter which is what appears to have happened here. Running multiple fibers likely costs more, ergo a place where OVH cut features (and redundancy) to have a lower price
- joepie91_ 11y agoAt OVH's scale, I doubt that this is a realistic cost issue. In fact, it is my understanding that OVH was already planning to bring more fiber connections online, and they've accelerated that process in order to get their network back up today. I don't know. It just irks me when people start making these kind of statements when they almost certainly do not have the data needed to support them. It smells of hype to me, especially where it concerns the 'hip' services like AWS, DigitalOcean, (previously) Linode, and so on. Trying to find some way to turn the narrative into something that puts those providers in a positive light, when it most likely had absolutely nothing to do with it. EDIT: And again, "it's cheaper so that must be why it has worse redundancy" is far, far too simplistic. There are many more variables at play.
- rstupek 11y agoWorse redundancy = cheaper is what I said, not the reverse. Amazon doesn't setup an AWS datacenter without multiple physically separated connections. The data to support my statement is in the fact they had 1 line connecting the data center and it was cut taking everything out. They may have started making additional lines into the center but they didn't have them today.
- ck2 11y agoI've seen "all of AWS down" reports around here a few times over the past few years. And then people list all the services that rely on it.
- ck2 11y agoFiber cuts are definitely not OVH's fault. However lack of additional routing might be. Apparently they were right in the middle of installing additional fiber on other routes for the datacenters but they are still dark and not online yet. They claim they are trying to accelerate getting those links online in their french ticket http://travaux.ovh.net/?do=details&id=15238 http://travaux.ovh.net/?do=details&id=15238
- ChuckMcM 11y agoReminds me of the joke: Traveller to ops guy, "Why do you have a piece of fiber optic cable in your rescue kit?", ops guy, "That way if I ever get stranded I just bury this cable nearby and wait for a back hoe to show up and cut it!" On a weird note, when evaluating data centers I had one tell me they had redundant connections to the Internet but what they really had were two lambdas on the same fiber. Sure its great that you have two providers but it is only one fiber didn't seem to sink in. Pass. The problem is very real of course, when a disgruntled AT&T engineer cut lines in South San Jose they did a huge amount of damage. Looking at their impact and the time it takes to resplice a fiber, a group of us calculated it would take no more than a 100 people to take California completely off the cabled Internet (satellite is really hard to disable from the ground). That seems pretty risky.
- ck2 11y agoThere are actually reports of people being spotted testing major connection points for fiber and power at various places (I think in California). Not sure if it is real or trumped up, but if real, it would be domestic terrorism for sure.
- nosuchthing 11y agoMaybe this what you're referring to: http://www.wsj.com/articles/SB10001424052702304851104579359141941621778 http://www.wsj.com/articles/SB100014240527023048511045793591... https://en.wikipedia.org/wiki/Metcalf_sniper_attack https://en.wikipedia.org/wiki/Metcalf_sniper_attack ?
- sithadmin 11y ago>when evaluating data centers I had one tell me they had redundant connections to the Internet but what they really had were two lambdas on the same fiber. Sadly, this kind of shenanigan is pretty common. I've also seen datacenters that have additional providers on different fiber...but when you follow the fibers upstream, all fiber winds up at the same POP. Predictably, outages at the POP would take the downstream datacenters offline.
- nikanj 11y ago
- fweespeech 11y agoNormally, I'd agree. However...OVH hasn't installed sufficient redundancy to safeguard against this stuff for years and its happened on an annual basis. http://status.ovh.net/?do=details&id=6629 http://status.ovh.net/?do=details&id=6629 http://status.ovh.net/?do=details&id=4376 http://status.ovh.net/?do=details&id=4376
- jedisct1 11y agoExcept what they completed today is the construction of another dark fiber network to this data center, going through a completely different physical path.
- fweespeech 11y agoAfter 5+ years? Finally? I'll believe it when it doesn't happen in 2016.
- chinathrow 11y agoI wonder, is this (still) common these days? Having a datacenter on a single physical cable?
- acveilleux 11y agoTo be fair, they're setting up the backup link. This DC, like many others, is well out of the way so running distinct fiber is lengthy process and even localizing fault and getting crews on site can be a drag. The location of Beauharnois probably being chosen because there's a Hydro plant there with 1903 MW of installed capacity.
- deepsun 11y agoAt Google -- no. At OVH -- yes, apparently. Google have it's own outages, caused mostly by a network config change or software update, but not from fiber cuts.
- tomschlick 11y agoPlus Google has a shit ton of redundancy. I remember watching a video about their site reliability team. Their job is to test worst case scenarios and they can/will cut an entire datacenter out of a service's (gmail, drive, etc) pool to see what happens.
- oasisbob 11y agoTheir exercises are also excellent at including the full organization in DiRT, not just engineering. eg, Just because someone is "authorized" to spend six-figures for scheduling a diesel delivery doesn't mean they will. _Weathering the Unexpected_ https://queue.acm.org/detail.cfm?id=2371516 https://queue.acm.org/detail.cfm?id=2371516
- imperialdrive 11y ago800gbps of connectivity cut! yikes, that would put all hands on deck... I'm hearing issues with Verizon/Terremark and Wellpoint networks from clients, wonder if it has anything to do with... godspeed OVH
- Tombar 11y agoLong time OVH customer here, they had a similar issue on May 2015 for the same DC in Canada where a car hit one of the poles along the BHS/MTL via the North route, damaging the optical fiber cable: 2 of the 3 pairs of cables where down. http://status.ovh.co.uk/?do=details&id=9603 http://status.ovh.co.uk/?do=details&id=9603 At the time, they told customers they were adding new fiber providers and cables.
- bryanlarsen 11y agoLaying new fiber takes years. They turned it on today, so it must have been in the works then too.
- Loic 11y agoThe effect on the weathermap is "clear cut": http://weathermap.ovh.net/#usa http://weathermap.ovh.net/#usa It looks like they still have one link running, but a small saturated one.
- ju-st 11y agoYes, technically the DC is not down... I can still ping forum.ovh.us and the webpage loads, but very slowly (as excpected).
- smoyer 11y agoThis is why you have fiber leaving your datacenter in two different direction and to two backbone providers. Once you do your due diligence, it's really hard to guarantee your providers never share infrastructure. Good luck guys!
- bcoates 11y agoAlso, be aware that fiber sabotage is a thing that happens. It's rare, but it's bad, and in my experience even top-notch providers take days to recover from it. Even if you aren't the target, it doesn't matter how many redundant links your single uplink provider has if the threat model is hostile actors with their infrastructure mapped out vs. wayward backhoes.
- oasisbob 11y agoTruth - stories of redundant paths getting groomed into the same cable years after provisioning seem to be all too common.
- jdalgetty 11y agoThis will certainly mean a temporary drop on botnet activity!!!
- napsterbr 11y agoFor customers of dedicated servers, remember to open a support ticket so you can get compensated accordingly. (default SLA = 5% per hour of downtime) https://twitter.com/rpurinton/status/661255883582775298 https://twitter.com/rpurinton/status/661255883582775298
- meirelles 11y agoThe OVH is great. A really good solution for many use cases. Of course you must know exactly what they are. Isn't a good idea trust a single DC setup, even worst if they are totally dirt-cheap. Unfortunately people misuse them.
- gesman 11y agoNow lets correlate this with "bad" TOR sites downtimes to narrow down their origins...
- skrowl 11y agoIt's back up as of 16:50 EST (UTC-5)!
- bdcravens 11y agoNot according to http://status.ovh.co.uk/?do=details&id=11304 http://status.ovh.co.uk/?do=details&id=11304
- napsterbr 11y agoActually, as you can see at their network map[1], the cables are still out. Only the three new ones, upgraded today, are working. [1] http://weathermap.ovh.net/#usa http://weathermap.ovh.net/#usa
- jedisct1 11y agoUp! http://status.ovh.co.uk/?do=details&id=9603 http://status.ovh.co.uk/?do=details&id=9603
- csense 11y agoLooks irrelevant, all comments seem to be from May.
- lacksconfidence 11y agoNotice the date of last update on that status: Thursday, 23 July 2015, 11:46AM
- Zarel 11y agoYou're looking at the wrong one. The downtime today is this one: http://status.ovh.co.uk/?do=details&id=11304 http://status.ovh.co.uk/?do=details&id=11304 And as of right now (4:17 PM CST), it says it's still being worked on: > We still don't have an ETA regarding the repairs on the existing fiber. > However, it's been confirmed that 2 out the 3 new pairs of the Eastern route (which were being setup) are now usable. We are working with our providers in order to get a full connection as quickly as possible: the only cross-connect linking OVH to our provider (in Montreal) is currently missing.
- jedisct1 11y agoWrong link, but everything is back to full speed no matter what :)
- jedisct1 11y agoConstructing a brand new dark fiber network over long distance requires an insane amount of work, plus a ton of legal and administrative issues to deal with. OVH had started working on the construction of a second dark fiber network a long time ago, with at least the capacity of the previous one. Due to a strange quirk of fate, they had planned to complete the construction within the next few days. After the fiber cut, they not only tried to locate and repair the cut pairs of cable, but also simultaneously rushed to finish the construction of the new network. They made in hours what was initially planned to require at least another week. The services are now back online at full speed, because they managed to finish installing the new cables, test them and connect the whole data center right away to the new network. And it worked.
- jagger27 11y agoThat's a story to tell your kids and grandkids. I have such great respect for network admins. I have no idea what it takes to get a network to route properly worldwide on multiple trunks.