3 ms·
I won't deny the role of luck, but I find that the more careful I am to keep my systems redundant, flexible, and rapidly deployable, the luckier I am. I did ha
by devishard 10y ago
I won't deny the role of luck, but I find that the more careful I am to keep my systems redundant, flexible, and rapidly deployable, the luckier I am.
I did have an issue once where a very large power outage that took out a data center. Total downtime was about 50 minutes, which consisted of:
1. Calling the hosting provider and ascertaining the problem.
2. Pulling the latest code onto the fallback server (from a different hosting provider).
3. Restoring the database from the backup server into a newly-created database server.
4. Changing the configs.
2 and 4 were able to happen while 3 was running.
As an aside, this is largely why specialized cloud services like AWS or AppEngine that promote vendor lock-in seem crazy to me. AWS can and does go down, and if your infrastructure is built around their tooling, you can't just provision a new box on a new provider.