3 ms·

Also when you decide on a backup, make sure your backup is not hosting their servers in the same DCs as is your primary, or worse yet that your backup is rentin

Also when you decide on a backup, make sure your backup is not hosting their servers in the same DCs as is your primary, or worse yet that your backup is renting their servers from the same people you rent your primary from.

jeremyjh 9y ago
Your backup site should be in a different region - e.g. if your primary is eastern US your backup should be central or western US. Otherwise you are vulnerable to various natural and man-made disasters.
vidarh 9y ago
My lesson in this was not servers, but networks. Back in the mid 90's at some point, Europe largely lost network connectivity to the US for about a day (this is all from recollection; I ran an ISP at the time, and we were affected, but I'm sure I've gotten the occasional detail wrong). Not entirely - there was significant capacity that was technically operational, but in practical terms large subsets of European networks were cut off. The reason was problem with one of Sprint's cables. It was carrying a lot of Nordunet traffic. Nordunet is/was the Nordic joint university network infrastructure. At the time, for historical reasons, Nordunet had more network capacity than most of the rest of Europe combined (consider that Norway was the second international connection to Arpanet in the early 70's due to the seismic array at Kjeller outside Oslo that provided essential data on Soviet nuclear tests; this made money flow to network research in Norway, and a joint Nordic effort let to significant investment in the other Nordic countries too; couple it with early work in Sweden, where they ended up hosting D-Gix - for many years the largest internet exchange point outside the US). So Sprints cable went down, and everyone started re-routing. Problem #1: Everyone had high capacity to D-Gix or other exchange point in the Nordic countries. These connections were now flooded by traffic from the Nordic networks where people were used to being able to saturate their 100Mbps network connections (it was such a downer starting my ISP, and going from 100Mbps at university to 512kbps aggregate outbound capacity at our offices). Except most of the links connecting to D-Gix where 34Mbps or less, and there were not that many of them, and they were connecting entire universities or entire countries... So connectivity within Europe started slowing down. Problem #2: Most of the links from elsewhere in Europe to the US were 8Mbps or less, with maybe a couple ones faster than that, and there were not that many of them. Certainly not enough to compensate for several hundred Mbps of lost capacity. Problem #3: Many of those links got so saturated and slow that failsafes were tripped and traffic started getting routed to their expensive backup links (e.g. people paying for a port and then paying for burst a 95th percentile basis and the like with expectation of normally not using them). Except, as it turns out, many of said backup links had been purchased from Sprint. Over that cable. Today the number of cables and overall capacity is vastly higher, with many more providers, so I don't think the same could happen again, and D-Gix is no longer as important as it was (though it's still one of the largest international interchange points), but as we sat there pinging and trying to figure out what was up, it was a vivid lesson in ensuring your backups are actually sufficiently separate from your main systems that they stand a chance of working.
- toast0 9y ago
  This type of outage was common in the US in the 90s as well. Many large (tier 1) ISPs found out that their redundant fiber links were not in redundant bundles, other tier 1s also had important links in the same bundles, and that a backhole had severed the bundle. Mostly, when this happened, west coast users couldn't contact east coast servers (and vice versa), but occasionally, BGP would work out, and packets would get back and forth, but go around the world.

jeremyjh 9y ago

Your backup site should be in a different region - e.g. if your primary is eastern US your backup should be central or western US. Otherwise you are vulnerable to various natural and man-made disasters.

vidarh 9y ago

My lesson in this was not servers, but networks. Back in the mid 90's at some point, Europe largely lost network connectivity to the US for about a day (this is all from recollection; I ran an ISP at the time, and we were affected, but I'm sure I've gotten the occasional detail wrong). Not entirely - there was significant capacity that was technically operational, but in practical terms large subsets of European networks were cut off. The reason was problem with one of Sprint's cables. It was carrying a lot of Nordunet traffic. Nordunet is/was the Nordic joint university network infrastructure. At the time, for historical reasons, Nordunet had more network capacity than most of the rest of Europe combined (consider that Norway was the second international connection to Arpanet in the early 70's due to the seismic array at Kjeller outside Oslo that provided essential data on Soviet nuclear tests; this made money flow to network research in Norway, and a joint Nordic effort let to significant investment in the other Nordic countries too; couple it with early work in Sweden, where they ended up hosting D-Gix - for many years the largest internet exchange point outside the US). So Sprints cable went down, and everyone started re-routing. Problem #1: Everyone had high capacity to D-Gix or other exchange point in the Nordic countries. These connections were now flooded by traffic from the Nordic networks where people were used to being able to saturate their 100Mbps network connections (it was such a downer starting my ISP, and going from 100Mbps at university to 512kbps aggregate outbound capacity at our offices). Except most of the links connecting to D-Gix where 34Mbps or less, and there were not that many of them, and they were connecting entire universities or entire countries... So connectivity within Europe started slowing down. Problem #2: Most of the links from elsewhere in Europe to the US were 8Mbps or less, with maybe a couple ones faster than that, and there were not that many of them. Certainly not enough to compensate for several hundred Mbps of lost capacity. Problem #3: Many of those links got so saturated and slow that failsafes were tripped and traffic started getting routed to their expensive backup links (e.g. people paying for a port and then paying for burst a 95th percentile basis and the like with expectation of normally not using them). Except, as it turns out, many of said backup links had been purchased from Sprint. Over that cable. Today the number of cables and overall capacity is vastly higher, with many more providers, so I don't think the same could happen again, and D-Gix is no longer as important as it was (though it's still one of the largest international interchange points), but as we sat there pinging and trying to figure out what was up, it was a vivid lesson in ensuring your backups are actually sufficiently separate from your main systems that they stand a chance of working.

toast0 9y ago
This type of outage was common in the US in the 90s as well. Many large (tier 1) ISPs found out that their redundant fiber links were not in redundant bundles, other tier 1s also had important links in the same bundles, and that a backhole had severed the bundle. Mostly, when this happened, west coast users couldn't contact east coast servers (and vice versa), but occasionally, BGP would work out, and packets would get back and forth, but go around the world.