Probably the biggest thing is to separate edge servers from backend servers, running DNS and reverse http(s) proxies (aka load balancers) on the edge servers. T

Probably the biggest thing is to separate edge servers from backend servers, running DNS and reverse http(s) proxies (aka load balancers) on the edge servers. This would allow you to achieve high availability on the internet facing layer and below. Use DNS routing and DNS failover to steer clients to specific edge servers and to avoid unavailable ones. The more edge servers you have from more independent hosting providers, the higher availability you can get. They can be cheap VPS servers. Failover between backends then can be handled on the edge servers on the reverse proxy layer. Backend servers won't even need physical IP addresses, they could tunnel to edge servers and be located anywhere. You would need to handle connectivity problems though, you can start by using fail fast approach with very low connect timeout and low timeouts and such to quickly and seemlessly switch between backends. I don't know of any off-the-shelf solution that can handle connectivity problems well, so at some point you will have to write your own tunneling proxy, but before that you can survive on nginx, it can mark backends as failed and avoid sending requests there for configurable amount of time. Start with at least 3 edge nodes from different hosting providers and 2 locations for backends, could be a single location if you can get 2 independent ISPs there.