4 ms·

Multihome your edge, folks.

api 11y ago
Glad to see this instead of bashing on OVH. This can happen anywhere. Every data center is at the end of one or maybe a few very thin little cables that can be cut very easily.
- deleted 11y ago
  [deleted]
  - joepie91_ 11y ago
    I don't really understand this reply. 1. What data are you comparing? AWS does not make network statistics public, and has had major entire-region outages in the past as well. 2. OVH is one of the biggest hosting companies in the world. What's the point of the "huge" adjective for AWS? 3. This really doesn't have anything to do with prices, per se. OVH is cheap because they automate everything heavily, not necessarily because their actual quality of service (aside from customer support) is worse. We don't have insight into the financials of either company, and such financials are typically very complex, so let's not make this into some pseudo-"you get what you pay for" argument.
    - rstupek 11y ago
      I think he's referring to AWS which sites their datacenters and have multiple fiber connections situated such that cutting one doesn't bring down the entire datacenter which is what appears to have happened here. Running multiple fibers likely costs more, ergo a place where OVH cut features (and redundancy) to have a lower price
      - joepie91_ 11y ago
        At OVH's scale, I doubt that this is a realistic cost issue. In fact, it is my understanding that OVH was already planning to bring more fiber connections online, and they've accelerated that process in order to get their network back up today. I don't know. It just irks me when people start making these kind of statements when they almost certainly do not have the data needed to support them. It smells of hype to me, especially where it concerns the 'hip' services like AWS, DigitalOcean, (previously) Linode, and so on. Trying to find some way to turn the narrative into something that puts those providers in a positive light, when it most likely had absolutely nothing to do with it. EDIT: And again, "it's cheaper so that must be why it has worse redundancy" is far, far too simplistic. There are many more variables at play.
        rstupek 11y ago
        Worse redundancy = cheaper is what I said, not the reverse. Amazon doesn't setup an AWS datacenter without multiple physically separated connections. The data to support my statement is in the fact they had 1 line connecting the data center and it was cut taking everything out. They may have started making additional lines into the center but they didn't have them today.
  - ck2 11y ago
    I've seen "all of AWS down" reports around here a few times over the past few years. And then people list all the services that rely on it.
- ck2 11y ago
  Fiber cuts are definitely not OVH's fault. However lack of additional routing might be. Apparently they were right in the middle of installing additional fiber on other routes for the datacenters but they are still dark and not online yet. They claim they are trying to accelerate getting those links online in their french ticket http://travaux.ovh.net/?do=details&id=15238 http://travaux.ovh.net/?do=details&id=15238
- ChuckMcM 11y ago
  Reminds me of the joke: Traveller to ops guy, "Why do you have a piece of fiber optic cable in your rescue kit?", ops guy, "That way if I ever get stranded I just bury this cable nearby and wait for a back hoe to show up and cut it!" On a weird note, when evaluating data centers I had one tell me they had redundant connections to the Internet but what they really had were two lambdas on the same fiber. Sure its great that you have two providers but it is only one fiber didn't seem to sink in. Pass. The problem is very real of course, when a disgruntled AT&T engineer cut lines in South San Jose they did a huge amount of damage. Looking at their impact and the time it takes to resplice a fiber, a group of us calculated it would take no more than a 100 people to take California completely off the cabled Internet (satellite is really hard to disable from the ground). That seems pretty risky.
  - ck2 11y ago
    There are actually reports of people being spotted testing major connection points for fiber and power at various places (I think in California). Not sure if it is real or trumped up, but if real, it would be domestic terrorism for sure.
    - nosuchthing 11y ago
      Maybe this what you're referring to: http://www.wsj.com/articles/SB10001424052702304851104579359141941621778 http://www.wsj.com/articles/SB100014240527023048511045793591... https://en.wikipedia.org/wiki/Metcalf_sniper_attack https://en.wikipedia.org/wiki/Metcalf_sniper_attack ?
  - sithadmin 11y ago
    >when evaluating data centers I had one tell me they had redundant connections to the Internet but what they really had were two lambdas on the same fiber. Sadly, this kind of shenanigan is pretty common. I've also seen datacenters that have additional providers on different fiber...but when you follow the fibers upstream, all fiber winds up at the same POP. Predictably, outages at the POP would take the downstream datacenters offline.
  - nikanj 11y ago
- fweespeech 11y ago
  Normally, I'd agree. However...OVH hasn't installed sufficient redundancy to safeguard against this stuff for years and its happened on an annual basis. http://status.ovh.net/?do=details&id=6629 http://status.ovh.net/?do=details&id=6629 http://status.ovh.net/?do=details&id=4376 http://status.ovh.net/?do=details&id=4376
  - jedisct1 11y ago
    Except what they completed today is the construction of another dark fiber network to this data center, going through a completely different physical path.
    - fweespeech 11y ago
      After 5+ years? Finally? I'll believe it when it doesn't happen in 2016.

api 11y ago

Glad to see this instead of bashing on OVH. This can happen anywhere. Every data center is at the end of one or maybe a few very thin little cables that can be cut very easily.

deleted 11y ago
[deleted]
- joepie91_ 11y ago
  I don't really understand this reply. 1. What data are you comparing? AWS does not make network statistics public, and has had major entire-region outages in the past as well. 2. OVH is one of the biggest hosting companies in the world. What's the point of the "huge" adjective for AWS? 3. This really doesn't have anything to do with prices, per se. OVH is cheap because they automate everything heavily, not necessarily because their actual quality of service (aside from customer support) is worse. We don't have insight into the financials of either company, and such financials are typically very complex, so let's not make this into some pseudo-"you get what you pay for" argument.
  - rstupek 11y ago
    I think he's referring to AWS which sites their datacenters and have multiple fiber connections situated such that cutting one doesn't bring down the entire datacenter which is what appears to have happened here. Running multiple fibers likely costs more, ergo a place where OVH cut features (and redundancy) to have a lower price
    - joepie91_ 11y ago
      At OVH's scale, I doubt that this is a realistic cost issue. In fact, it is my understanding that OVH was already planning to bring more fiber connections online, and they've accelerated that process in order to get their network back up today. I don't know. It just irks me when people start making these kind of statements when they almost certainly do not have the data needed to support them. It smells of hype to me, especially where it concerns the 'hip' services like AWS, DigitalOcean, (previously) Linode, and so on. Trying to find some way to turn the narrative into something that puts those providers in a positive light, when it most likely had absolutely nothing to do with it. EDIT: And again, "it's cheaper so that must be why it has worse redundancy" is far, far too simplistic. There are many more variables at play.
      - rstupek 11y ago
        Worse redundancy = cheaper is what I said, not the reverse. Amazon doesn't setup an AWS datacenter without multiple physically separated connections. The data to support my statement is in the fact they had 1 line connecting the data center and it was cut taking everything out. They may have started making additional lines into the center but they didn't have them today.
- ck2 11y ago
  I've seen "all of AWS down" reports around here a few times over the past few years. And then people list all the services that rely on it.
ck2 11y ago
Fiber cuts are definitely not OVH's fault. However lack of additional routing might be. Apparently they were right in the middle of installing additional fiber on other routes for the datacenters but they are still dark and not online yet. They claim they are trying to accelerate getting those links online in their french ticket http://travaux.ovh.net/?do=details&id=15238 http://travaux.ovh.net/?do=details&id=15238
ChuckMcM 11y ago
Reminds me of the joke: Traveller to ops guy, "Why do you have a piece of fiber optic cable in your rescue kit?", ops guy, "That way if I ever get stranded I just bury this cable nearby and wait for a back hoe to show up and cut it!" On a weird note, when evaluating data centers I had one tell me they had redundant connections to the Internet but what they really had were two lambdas on the same fiber. Sure its great that you have two providers but it is only one fiber didn't seem to sink in. Pass. The problem is very real of course, when a disgruntled AT&T engineer cut lines in South San Jose they did a huge amount of damage. Looking at their impact and the time it takes to resplice a fiber, a group of us calculated it would take no more than a 100 people to take California completely off the cabled Internet (satellite is really hard to disable from the ground). That seems pretty risky.
- ck2 11y ago
  There are actually reports of people being spotted testing major connection points for fiber and power at various places (I think in California). Not sure if it is real or trumped up, but if real, it would be domestic terrorism for sure.
  - nosuchthing 11y ago
    Maybe this what you're referring to: http://www.wsj.com/articles/SB10001424052702304851104579359141941621778 http://www.wsj.com/articles/SB100014240527023048511045793591... https://en.wikipedia.org/wiki/Metcalf_sniper_attack https://en.wikipedia.org/wiki/Metcalf_sniper_attack ?
- sithadmin 11y ago
  >when evaluating data centers I had one tell me they had redundant connections to the Internet but what they really had were two lambdas on the same fiber. Sadly, this kind of shenanigan is pretty common. I've also seen datacenters that have additional providers on different fiber...but when you follow the fibers upstream, all fiber winds up at the same POP. Predictably, outages at the POP would take the downstream datacenters offline.
- nikanj 11y ago
fweespeech 11y ago
Normally, I'd agree. However...OVH hasn't installed sufficient redundancy to safeguard against this stuff for years and its happened on an annual basis. http://status.ovh.net/?do=details&id=6629 http://status.ovh.net/?do=details&id=6629 http://status.ovh.net/?do=details&id=4376 http://status.ovh.net/?do=details&id=4376
- jedisct1 11y ago
  Except what they completed today is the construction of another dark fiber network to this data center, going through a completely different physical path.
  - fweespeech 11y ago
    After 5+ years? Finally? I'll believe it when it doesn't happen in 2016.