4 ms·
The entire point is that "an honest mistake" shouldn't be able to cause a large incident at a mature company. Requiring tests, code reviews, a test env, locked-
by Shaanie 3y ago
The entire point is that "an honest mistake" shouldn't be able to cause a large incident at a mature company. Requiring tests, code reviews, a test env, locked-down prod etc may slow down delivery, but it also prevents Andrew the sales dev accidentally corrupting the production database when he's doing a "quick fix for a customer".
- volkk 3y agoi don't necessarily mean prod DB going down as an "honest mistake," though. I think that is pretty deserving of a postmortem. What i meant is like, a bug making it through that stops something critical from working because the dev didn't write a test for it. We already know the resolution, we know who fucked up, we understand why it happened. There's no need for a postmortem to painstakingly add another layer or process for devs that screw up. Like, a lot of companies will then say "okay how can we prevent developers from forgetting to write tests" and start thinking about auto-generators for code, or adding some CI check, or a level of bureaucracy when all that was needed was the dev to own up to the mistake, or maybe their manager to tell them "c'mon, this can't happen again" and we can all move on with our day. And, if it keeps happening across differing folks, only THEN would I start thinking about process additions.
- red-iron-pine 3y agorisk of outage + its costs vs. risk of mitigation + its costs. or as fight club said > Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one.