3 ms·
How many memory leaks were discovered only during the winter code freeze, because there were no pushes being done, so no server restarts
by rolisz 3y ago
How many memory leaks were discovered only during the winter code freeze, because there were no pushes being done, so no server restarts
- jedberg 3y agoAt reddit we would randomly select a process to kill every 10 minutes out of the 10 or so on each machine, just so they would all get a restart in case we didn't do a deployment for a few days. At Amazon they schedule service bounces during code freeze for any service that is known to have memory leaks because it's easier than finding the leak, which isn't usually an issue since it gets deployed so often.
- yjftsjthsd-h 3y agoAnd as a nice bonus you get chaos monkey for free:)
- calvinmorrison 3y agoAt Fastmail the ops team we ran fail overs all the time just to get our failures so reliable they worked no matter what. Only once in my tenure did a fail over fail and in that case there was a --yolo flag