4 ms·
My experience is: Downtime does matter. Customers ignore downtime when you're a service with big lock-in or no real alternative (GitHub, Facebook, etc). Becaus
by nh2 3y ago
My experience is: Downtime does matter.
Customers ignore downtime when you're a service with big lock-in or no real alternative (GitHub, Facebook, etc). Because then they can't do anything about it.
When you're a small/medium service, and you're down for 3 hours during a time where 4 employees of a company need you for productive work, they ask what's up, and if they must expect that this will happen again. Generally, shorter, one-off downtimes are forgiven, but longer or recurring ones are not.
We run on Hetzner dedicated, and I think the best way to run most sensible software is to run on 3 or 5 beefy machines. If you only have 1 machine, the following will take you down, all of which have happened in my production systems (except the last, which I only read about in the news for 3 hosters):
* You need to reboot for kernel security updates. A reboot causes 3 minutes downtime because that's how long a reboot takes on server hardware, including Hetzner's (multiple POST style screens, netboot timeouts and so on, most of which make sense). Kernel updates come in approximately every 3 weeks. You may skip some of them that are not security-relevant for you, but this requires careful analysis, which is more effort than just installing.
* The mainboard dies. It needs to be replaced. Immediately causes 3 hours of downtime.
* Some disk or RAM dies in a way that prevents the mainboard from booting. This should not happen, but it does. Also 3 hours of downtime.
* One of these happens at 1 am on Saturday. You are in Europe or Asia. If you are not set up to be paged out of bed, your US customer will see 8 hours of downtime.
* The top-of-rack or datacenter building core router has some trouble, and it takes the hoster 3 hours to fix it. 3 hours of downtime.
* There's a fire or sprinkler flooding in your data center. It takes the hoster 2 weeks to clean up the physical mess. 2 weeks of downtime, unless you manage to re-build the infra somewhere else and restore from backups, then maybe 48 hours of downtime.
Note most of this also applies if you run a single VM in AWS. If you're lucky, the restore times will be faster by being a VM. But that is not guaranteed; I've had instances not do their work for some hours and afterwards got the email "We noticed a problem with the physical host of your VM, we've automatically migrated it somewhere else now".
Most of these problems go away if you have 3 servers (of the type the original post describes) instead of one, which is only slightly more effort.
- floating-io 3y agoI largely agree with this. I didn't take "one server" literally; more as hyperbole. Concern about availability is overblown, not irrelevant! There are shades of gray, though. Most sites don't need to engineer their system to hide catastrophic database failures from the user. Not being able to tolerate a single web server failure, though... that's just careless, and will end up giving you a bad reputation. You probably don't actually need five nines -- but three probably won't go amiss...