4 ms·
Y'all don't do post-mortem investigations / action items? I get the desire to troubleshoot but priority 0 is make the system functional for users again, litera
by Spivak 3y ago
Y'all don't do post-mortem investigations / action items?
I get the desire to troubleshoot but priority 0 is make the system functional for users again, literally everything else can wait. I once had to deal with an outage that required we kill all our app servers every 20 minutes (staggered of course) because of a memory leak while it was being investigated.
- Salgat 3y agoUsually depends on the impact. If it's one of many instances behind a load balancer and was easily fixed with no obvious causes, then we move on. If it happens again, we have a known short-term fix and now we have a justified reason to devote man-hours to investigating and doing a post-mortem.
- kqr 3y ago> I get the desire to troubleshoot but priority 0 is make the system functional for users again, literally everything else can wait. What numbers went into this calculation, to get such an extreme result as concluding that getting it up again is always the first priority? When I tried to estimate the cost and benefit, I have been surprised to make the opposite conclusion multiple times. We ended up essentially in the situation of "Yeah, sure, you can reproduce the outage in production. Learn as much as you possibly can and restore service after an hour." This is in fact the reason I prefer to keep some margin in the SLO budget -- it makes it easier to allow troubleshooting an outage in the hot path, and it frontloads some of that difficult decision.