3 ms·
There's a reason the military does drills. Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your system
by rkagerer 6y ago
There's a reason the military does drills.
Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems. It's especially critical for disaster recovery (a DR plan that's never tested isn't worth the paper it's written on).
Consider going one step further and randomly injecting artificial failures into production shards, so handling them becomes a regular affair. When you build out monitoring, that might be a good time to think about the kind of stimulus that would be effective at exercising those monitors/metrics/alerts (think unit testing analogy). You can automate an evil adversary bot, or have humans do Red team / Blue team challenges. Yes, you're limited how much you can break without too severely impacting production, but if you've designed enough redundancy you should be able to achieve reasonable coverage. The more you can engineer simulated failures into your regular workflow, the less of a big deal it'll be when real ones occur.
Instead of watching the watchers, engage them with stimulus that keeps them sharp and prevents them from getting bored.
- throwaway0a5e 6y ago>Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems. All those man hours cost money. Is it worth it? Depends on how much downtime costs you.
- WJW 6y agoMost companies don't know how much downtime costs them and from their behaviour seem to treat it either as $0/hr or $infinite/hr. To be fair, it's difficult to estimate correctly. Apart from the direct loss of income there is also reputation loss, possibly reduced signups and increased churn, etc. It's not at all constant either, one outage per 10 years will give you a radically different reputation than one outage per week.
- Ntrails 6y agoIt's also working out which outages matter the most. A 10 day outage is not equivalent to 10 1 day outages, the latter has a multitude of important reputational risks - but the former is quite possibly enough to end the business completely.
- aboringusername 6y agoDepends on how you value the potential outcomes. A while ago, I think it was Gitlab that didn't check their backups and they were a dud. That caused quite a few problems...How much is it worth to them after the fact? I'd say a fair bit. But hindsight...
- dannyw 6y agoIf you run a fairly discretionary web app, like perhaps a game where downtime will cost you lost revenue for the time period but that's it, it's probably not worth it. If you are running a communications platform like Slack or Gmail, it's hella worth it.