6 ms·

Reminds me a lot of working in tech to be honest. Things break, postmortems are held. Even if the breakage is as simple as "this was an honest mistake", everyth

by volkk 3y ago

Reminds me a lot of working in tech to be honest. Things break, postmortems are held. Even if the breakage is as simple as "this was an honest mistake", everything is held to "how can this never happen again," so process and red tape is added. Rinse and repeat ad nauseam until your delivery speed is butchered, the bugs _still_ happen, but hey, a project manager or someone somewhere now gets to pat themselves on the back for running a meeting and adding more process that they can now tout on their future interviews.

jonhohle 3y ago
But at least that exact same bug can never happen again!
londons_explore 3y ago
This is fixed by putting a dollar number on outages, and a seperate dollar number on process. Ie. "if each release has an extra day of tests and checks, then it will delay our hockey stick growth curve by 10%. Thats worth $X". Turns out, it's usually better to be fast and break things, over trying to be reliable.
- slowmovintarget 3y ago
  Up until the point where part of what you're selling is reliability. "Use our futzel service, you will be able to futz around all day long, and it won't go down!"
- LawTalkingGuy 3y ago
  > Turns out, it's usually better to be fast and break things In terms of # of correct guesses, sure. In terms of damage from guessing incorrectly over time, usually not. The trick is to separate processes - A CSS update probably isn't very risky, and an auth or login update probably is. Don't bundle them into "website updates". Run ahead with the layout changes and spend some time on failure-planning for the auth change.
  - shadowgovt 3y ago
    Precisely this. The "move fast and break things" mantra came from a context where there were no users for the product yet and the people most inconvenienced by breakage are your own. It's a mantra saying "Don't worry about screwing up the UI team with a DB migration if it takes only five minutes to reverse the migration or ten minutes to tweak the UI; you can hash it out together and get back to work on the real problem." It's a reminder that exposing yourself to a little more pain now gets you to viable product faster, and the faster a startup gets to viable product, the likelier it is to take off before it hits the end of its runway. It is not to be applied to destructive changes to systems users care about, and none of the companies named in the 2017 book by that title use that engineering tactic on their flagship products because real people care now.
Shaanie 3y ago
The entire point is that "an honest mistake" shouldn't be able to cause a large incident at a mature company. Requiring tests, code reviews, a test env, locked-down prod etc may slow down delivery, but it also prevents Andrew the sales dev accidentally corrupting the production database when he's doing a "quick fix for a customer".
- volkk 3y ago
  i don't necessarily mean prod DB going down as an "honest mistake," though. I think that is pretty deserving of a postmortem. What i meant is like, a bug making it through that stops something critical from working because the dev didn't write a test for it. We already know the resolution, we know who fucked up, we understand why it happened. There's no need for a postmortem to painstakingly add another layer or process for devs that screw up. Like, a lot of companies will then say "okay how can we prevent developers from forgetting to write tests" and start thinking about auto-generators for code, or adding some CI check, or a level of bureaucracy when all that was needed was the dev to own up to the mistake, or maybe their manager to tell them "c'mon, this can't happen again" and we can all move on with our day. And, if it keeps happening across differing folks, only THEN would I start thinking about process additions.
- red-iron-pine 3y ago
  risk of outage + its costs vs. risk of mitigation + its costs. or as fight club said > Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one.
j-bos 3y ago
I believe DHH of Basecamp(?) fame calls this scarring on the first cut.

by volkk 3y ago

jonhohle 3y ago

But at least that exact same bug can never happen again!

londons_explore 3y ago

This is fixed by putting a dollar number on outages, and a seperate dollar number on process. Ie. "if each release has an extra day of tests and checks, then it will delay our hockey stick growth curve by 10%. Thats worth $X". Turns out, it's usually better to be fast and break things, over trying to be reliable.

slowmovintarget 3y ago
Up until the point where part of what you're selling is reliability. "Use our futzel service, you will be able to futz around all day long, and it won't go down!"
LawTalkingGuy 3y ago
> Turns out, it's usually better to be fast and break things In terms of # of correct guesses, sure. In terms of damage from guessing incorrectly over time, usually not. The trick is to separate processes - A CSS update probably isn't very risky, and an auth or login update probably is. Don't bundle them into "website updates". Run ahead with the layout changes and spend some time on failure-planning for the auth change.
- shadowgovt 3y ago
  Precisely this. The "move fast and break things" mantra came from a context where there were no users for the product yet and the people most inconvenienced by breakage are your own. It's a mantra saying "Don't worry about screwing up the UI team with a DB migration if it takes only five minutes to reverse the migration or ten minutes to tweak the UI; you can hash it out together and get back to work on the real problem." It's a reminder that exposing yourself to a little more pain now gets you to viable product faster, and the faster a startup gets to viable product, the likelier it is to take off before it hits the end of its runway. It is not to be applied to destructive changes to systems users care about, and none of the companies named in the 2017 book by that title use that engineering tactic on their flagship products because real people care now.

Shaanie 3y ago

The entire point is that "an honest mistake" shouldn't be able to cause a large incident at a mature company. Requiring tests, code reviews, a test env, locked-down prod etc may slow down delivery, but it also prevents Andrew the sales dev accidentally corrupting the production database when he's doing a "quick fix for a customer".

volkk 3y ago
i don't necessarily mean prod DB going down as an "honest mistake," though. I think that is pretty deserving of a postmortem. What i meant is like, a bug making it through that stops something critical from working because the dev didn't write a test for it. We already know the resolution, we know who fucked up, we understand why it happened. There's no need for a postmortem to painstakingly add another layer or process for devs that screw up. Like, a lot of companies will then say "okay how can we prevent developers from forgetting to write tests" and start thinking about auto-generators for code, or adding some CI check, or a level of bureaucracy when all that was needed was the dev to own up to the mistake, or maybe their manager to tell them "c'mon, this can't happen again" and we can all move on with our day. And, if it keeps happening across differing folks, only THEN would I start thinking about process additions.
red-iron-pine 3y ago
risk of outage + its costs vs. risk of mitigation + its costs. or as fight club said > Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one.

j-bos 3y ago

I believe DHH of Basecamp(?) fame calls this scarring on the first cut.