3 ms·
To a non-engineering manager or non-SRE, this may seem absurd. We do these in a more planned way where I work and call them "fire drills". Everything prod sho
by sacnoradhq 4y ago
To a non-engineering manager or non-SRE, this may seem absurd.
We do these in a more planned way where I work and call them "fire drills".
Everything prod should be HA and evict-able from a particular server, rack, and/or DC floor and survive. If it's not, it's a sign something is wrong.
The goal isn't fun, it's to find corner cases and places that aren't HA by meatcloud, chaos monkey-style.
Ideally, lights-out PDUs and Ethernet switch port control can simulate "pulling plugs" in an automated way and do so remotely and more cheaply.
If you bought a company, maybe you'd do this once to get a baseline, but with the express intention of not doing it for fun or to be a jerk.