3 ms·
wrong lesson. The right lesson is "have more than one data center, use automatic failover"
by lsc 17y ago
wrong lesson.
The right lesson is "have more than one data center, use automatic failover"
- lsc 17y agoto be clear, the above is meant as the lesson to the customer, not rackspace. It would be very nearly impossible for rackspace to make their virtual servers stay up if a whole data center went down. It's quite difficult to keep a VPS up when even just the hardware it is on goes down.
- ars 17y agoHow? You need to put a switchover device somewhere. And that device (or NOC, or power, or router feeding it), can fail. Unless you want to give you dns addresses a 10 second ttl, and update them as needed. Really, how? I would love to do this, but I can't think of any method that does not have a single point of failure somewhere. (And I'm not as big as google with my own network lines, something that a regular budget can do.)
- lsc 17y agoBGP. the idea is that you could have a data center in San Jose, connected via he.net, and another in, say, phoenix, connected via Cogent. you announce the same /24 out each data center... if one goes down, it stops announcing and all traffic goes to the other data center. The problem is that it's not a cheap answer, and that your application needs to be able to deal with queries going to either data center at any time. another option I've seen people use which is not nearly as robust but still 'automatic' is to have two authoritative nameservers; one at each location. But the nameserver at location 1 resolves all A records to IPs at location 1. At location 2, the nameserver is configured to resolve all A records to IP addresses at location 2. (of course, this has all the dns caching issues as setting the TTL to 10 minutes) A third option would be to have some sort of redirector on a network that is made redundant via BGP, then have some sort of hook so you can change the target of the redirector. But I don't know of anyone offering that service for cheap.
- mseebach 17y agoNo, the right lesson is to take a deep breath and ask yourself what three hours of downtime is worth to your business and compare that to the cost of full failover. The cost of full failover is first off a doubling of your hardware and datacenter costs. Then add the cost of developing a software system that can handle seamless failover, as well as testing it. By this I also mean the cost of significantly slower release-cycles, because you're not going to release a new feature without testing if it breaks the fail-over, are you? If this comes out in favor of doing full failover, you're a bank or an airline or similar. You're probably not even Google or Facebook. Chances are that you're not taking infrastructure advice from an online discussion thread.
- streety 17y agoAlthough I think your overall point is valid I wonder whether there is a middle ground on the hardware. For many purposes offering a degraded service would be better than nothing and could perhaps be handled with 1.5x your needed resources rather than 2x. It may also be possible to make use of this fabled cloud to take the 0.75x resources you have left after an outage back up to 1x. All this does add development time though and, as you say, for most sites it isn't going to be worth it.
- lsc 17y agoInfrastructure, for most businesses, is a vanishingly small percentage of the total costs of doing business. Hell, I sell infrastructure, and at 750 customers, my infrastructure costs are still less than I'd pay for my time if I was paying myself market rate. But you do have a good point, that you do need to weigh the cost of a few hours downtime every now and again with the cost of putting yourself in a position where you can avoid such downtime, because yes, sometimes avoiding the downtime is more costly than just taking the hit. I think at a minimum, though, you need to have off-site backups, and the ability to restore to a new provider if your first provider has a HyperVM level disaster that kills all your data and all the backups at that provider.