4 ms·
> 2:42 AM PDT / 7:42 UTC: Our on-call engineers restarted the redis-master to address the high load. Over the years I've learned that restarting something that
by pilif 13y ago
> 2:42 AM PDT / 7:42 UTC: Our on-call engineers restarted the redis-master to address the high load.
Over the years I've learned that restarting something that's under high load is not the correct solution to the problem.
By restarting the thing under load, you will always further increase the load when the thing you've restarted is coming back. Recovery will take longer and stuff could fail even more badly.
Of course there's the slim possibility that the thing under load has actually crashed or entered some infinite loop, but over the years I've learned that it's far more likely to be my fault than their fault; the abnormal load is far more likely caused by a problem at my end than a bug in the software that's misbehaving.
I know that just restarting is tempting, but I've seen so many times that spending the extra effort to analyze the root case usually is worth it.
- RobSpectre 13y agoIt is indeed. Proper procedure would be to pivot to a slave. We have reconfigured all our Redis hosts to prohibit a restart on masters to prevent exactly that temptation.
- pilif 13y agoYeah. I totally agree with your conclusions in your article. I was making a general remark because this doesn't just apply to you guys but to everyone else too :-) Btw: When going to a slave, be careful because the state the slave is in might be out of date compared to overloaded master at the point of moving over.
- aphyr 13y agoI should mention that promoting a secondary to a primary in an asynchronously replicated system can (and almost certainly will, under load) result in the loss of committed transactions. Redis has two resynchronization modes. One just recovers a small delta from the primary, if it hasn't diverged that far. If the primary has accepted too many writes, the secondary has to initiate a full resync, which is much more expensive. Since Twilio's postmortem says their nodes initiate full resyncs, I suspect that had Twilio promoted a secondary, they would have lost a significant number of writes. Probably best, under these kinds of scenarios, to design the system such that lost writes don't cause overbilling. ;-)
- ars 13y agoYup. Came here to say the same thing as soon as I saw that. Never restart a service that is under high load. Block the network if you must to try to reduce the load, but don't restart. Part of the problem of being under high load is that sometimes data does not get written out to disk like it should (because of the load on the OS), so restarting can easily corrupt things - especially if you restart the computer rather than the service. The other takeaway is reboot all your servers monthly! (Or after any changes.) That way you can be sure they'll come back up properly. Reboot when you have time to work on it, so that you don't have an emergency at the worst possible time.