Sometimes you have trouble reproducing an issue outside of prod, even with things like tcpreplay or blkreplay. You could just kill off a problematic instance, b

Sometimes you have trouble reproducing an issue outside of prod, even with things like tcpreplay or blkreplay. You could just kill off a problematic instance, but then you have trouble knowing why there was a problem to begin with. Grey failures might not be obvious in logs or metrics. The idea that you never ever have to SSH into a production server is a nice ideal, but I've never seen it survive reality unless you just shrug about issues occurring and don't mind not being able to root cause them.