3 ms·
> In my entire 10+ year career I've never had a service I was working on have a 20 hour outage. Through luck or careful planning and good resources? Once you
by gregmac 10y ago
> In my entire 10+ year career I've never had a service I was working on have a 20 hour outage.
Through luck or careful planning and good resources?
Once you have enough hardware to handle load, it all really comes down to risk mitigation.
Assuming it handles load, your service could probably run on a single server, with a single drive, on a DSL connection for years without any issue. I would say if that happens, it has less to do with your skill as an operator and more to do with the luck of not having a power outage, drive failure or backhoe operator dig up a line.
Can your service withstand your office/datacenter burning down? Or your datacenter being shutdown with all assets seized by the SEC (true story)? Or a massive earthquake or flood or hurricane that disrupts the entire city?
- devishard 10y agoI won't deny the role of luck, but I find that the more careful I am to keep my systems redundant, flexible, and rapidly deployable, the luckier I am. I did have an issue once where a very large power outage that took out a data center. Total downtime was about 50 minutes, which consisted of: 1. Calling the hosting provider and ascertaining the problem. 2. Pulling the latest code onto the fallback server (from a different hosting provider). 3. Restoring the database from the backup server into a newly-created database server. 4. Changing the configs. 2 and 4 were able to happen while 3 was running. As an aside, this is largely why specialized cloud services like AWS or AppEngine that promote vendor lock-in seem crazy to me. AWS can and does go down, and if your infrastructure is built around their tooling, you can't just provision a new box on a new provider.