4 ms·
An occasional outage is sometimes good for an app (depending, of course, on how mission-critical it is): 1. People don't realize how much they love and depend
by jen_h 7y ago
An occasional outage is sometimes good for an app (depending, of course, on how mission-critical it is):
1. People don't realize how much they love and depend on you until you're gone.
2. Keeps you on your toes, it's easy to get complacent when everything just runs along happily for months and years on end.
I do wish there was a way to train users that millions of them reloading constantly as service ramps back up doesn't accelerate the ramp-up time, though. ;)
- cle 7y agoYes! If you haven’t had an operational issue with a service in a while, you should force one on a testing to stack to make sure it fails like you expect, you can recover gracefully, your monitors work, etc. (lots of folks call these “game days”). A service that hasn’t had an ops issue in a long time is a ticking time bomb: when it does fail, nobody will be very familiar with it, and environmental assumptions might have changed, which could result in taking a much longer time to recover. When I design services these days, I try to design them so these failure scenarios are constantly exercised. Eg if I care about multi-AZ resiliency, I try to design it so that it’s forced to fail over to other AZs all the time. Or at the least, write tests for the scenario. Exceptional behavior or code paths are dangerous.