6 ms·

As I get older, I find myself enjoying these types of stories less and less. My issue comes from the fact that nobody seems comfortable having a conversation ab

As I get older, I find myself enjoying these types of stories less and less. My issue comes from the fact that nobody seems comfortable having a conversation about facts and data, instead resorting to childish analogies about turning knobs. That’s not how our jobs work. We don’t “adjust a carefulness meter.” We make conscious choices day to day based on the work we’re doing and past experience. As an EM, I’d be very disappointed if the incident post mortem was reduced to “your team needs to be more careful.” What I want from a post mortem is to know how we could prevent, detect or mitigate similar incidents in future and to make those changes to code or process. We then need to lean on data and experience of what the trade offs of those changes would be. Asking a test? Go for it. Adding extra layers of approval before shipping? I’ll need to see some very strong reasons for that.

Aeolun 2y ago
> What I want from a post mortem is to know how we could prevent, detect or mitigate similar incidents in future and to make those changes to code or process. The answer this post gives to that bizarre question that always gets asked, is ‘nothing’, unless you want to significantly adjust the speed that we deliver features. Any added process or check is going to impose overhead and make the team a little bit less happy. Ocassionally you’ll have a unicorn situation where there is actually a relatively simple fix, but those are few and far between. In extremis, you’re reduced to a situation in which you have zero incidents, but you also have zero work getting done.
- hamandcheese 2y ago
  > Ocassionally you’ll have a unicorn situation where there is actually a relatively simple fix, but those are few and far between. Perhaps we have different backgrounds, but even in late stage startups I find there is an abundance of low hanging fruit and simple fixes. I'm sure it's different at Google, though.
- scott_w 2y ago
  That’s simply not true. Some processes are good, some post-mortem outcomes will focus on improving deployment speed (so you can revert changes faster), or improve your monitors so you can detect (and mitigate) incidents faster. On the other hand, enforcing a manual external QA check on every release WILL slow things down. You’re repeating the same mistake as the article by assuming “process” sits on a grade that naturally slows work down. This is because you’re not being precise in your reasoning. Look at the specifics and make a decision based on the facts in front of you.
  - braza 2y ago
    > This is because you’re not being precise in your reasoning. Look at the specifics and make a decision based on the facts in front of you I agree with the premise here, but in my experience running incidents review the issue that I see it’s a mixture of a performatic safetycism with reactivity. Processes are the cheap bandaid to fix design, architectural and cultural issues. Most of the net positive micro-reforms that we had after incident reviews were the ones that invested in safety nets, faster recoveries, and guardrailing than a new process and will tax everyone.
    - scott_w 2y ago
      > Processes are the cheap bandaid to fix design, architectural and cultural issues. They can be, yes. I have a friend that thinks I'm totally insane by wanting to release code to production multiple times a day. His sweet spot is once every 2 weeks because he wants QA to check over every change. Most of his employers can manage once a month at best, and once a quarter is more typical. > Most of the net positive micro-reforms that we had after incident reviews were the ones that invested in safety nets, faster recoveries, and guardrailing than a new process and will tax everyone. I 100% agree with this. Your comment also reminded me to say that incident reviews are necessary but not sufficient. You also need engineering leadership reviewing at a higher-level to make bigger organisational or technical changes to further improve things.
shubhamjain 2y ago
Thanks for saying this. This mindset has infiltrated all the engineering teams and made software development hell for those of us who actually like shipping. Being more careful, adding more checks and processes, has exponentially less returns (just invert the graph). Somehow though, teams have been lead to believe that each incident needs to be responded with more processes.
- nicbou 2y ago
  This is an even bigger problem putside of our profession. Organisations do everything in their power to reduce the agency of employees and the general public through process. A company would rather spend 20 hours verifying a purchase than allow an unnecessary purchase every now and then. German culture in particular seems to favour this to an extreme.
  - braza 2y ago
    I like the German workplace (to white collar job) for several reasons but this one is one that drives me away. We used to have an issue of deployments breaking in production, and one of the reasons was that we did not have some kind of smoke test in the post deployment (in our case we only had a rolling update as a strategy). The rational solution was only create that post-deployment step. The solution that our German managers demanded: cut access to deployment for the entire team, “on-call” to check the deployments, and a deployment spreadsheet to track it.
    - intelVISA 2y ago
      Good lord, are all those non-tech managers the reason Europe can't seem to build viable tech companies?
      - nicbou 2y ago
        In my experience it’s a cultural problem in Germany. Everything has to be done methodically, even if the method adds a disproportionate amount of friction. Often, the purported benefits are not even there. The thoroughness is full of holes, the diligence never done, the follow-ups never happening. It leads to situations where you need a certificate from your landlord that you take to the certified locksmith that your landlord contracted and show a piece of ID to order a key double that arrives 3 business days later at a cost of 60€. A smart German knows that there’s a locksmith in the basement of a nearby shopping mall that will gladly duplicate any key without a fuss, but even then the price is inflated by the authorised locksmiths. I document German bureaucracy for immigrants. Everything is like this. Every time I think “it can’t really be this ridiculous, I’m being uncharitable”, a colleague has a story that confirms that the truth is even more absurd. It’s funny until you realise the cost it has for society at large. All the wasted labour, all the bottlenecks, and little to show for it.
hamandcheese 2y ago
I get where you are coming from, and I certainly do expect actual facts, data, and reasoning to be a part of any serious postmortem analysis. But those will almost always be in relation to a very specific circumstance. I think there is still room for generalized parables such as this article - otherwise, we would be reading a postmortem blog post, which are also common here and usually do contain what you are asking for.
- scott_w 2y ago
  I think you can generalise without resorting to silly games like the article does. I gave some examples in a sibling comment that are high level enough to give an idea of the types of things I’d think about, without locking in to a specific incident I was part of.
odo1242 2y ago
Isn't that the point of the story though? To say "we need to have a conversation about facts, rather than something like always saying "we need to be more careful next time" when there's a problem? As far as I can tell, the author doesn't really give any generalized advice on how careful you should be, he's just pointing at the "carefulness dial" and telling people to make an informed decision.
- scott_w 2y ago
  I'm not really sure it is the author's point. I re-read the article to try and find your interpretation and I couldn't really find it there. Maybe a slight hint in the coda?
1oooqooq 2y ago
you're probably on the spectrum, as most anyone here. there's a tradeoff on shipping garbage fast which won't explode on your hands and getting promoted. and there's also the political art of selling what you want to other people by masking it as what they want. you and i and most people here will never understand any of that. good luck. people who do understand will have the careless knob stuck at 11. these analogies help us rational people point out the BS at least, without having to fight the BSer.
- scott_w 2y ago
  > you're probably on the spectrum, as most anyone here. Your comment starts with an ableist slur so I’m sure it’s going to be good /s > you and i and most people here will never understand any of that. good luck. people who do understand will have the careless knob stuck at 11. Nah, reading this comment wasn’t worthwhile after all. > these analogies help us rational people point out the BS at least, without having to fight the BSer. How cute, you think you’re “rational.”
astura 2y ago
Exactly, these stories only appeal to children and childish adults.
- scott_w 2y ago
  I think we can criticise the analogy as reductive without insulting people, here.
- gffrd 2y ago
  Or maybe put more accurately: these stories don’t appeal to you and people like you. What is your objection? Oversimplification? Lack of utility?
picometer 2y ago
Agreed. I recall being taught in college physics labs: there is no such thing as “human error”. Instead, think about the causes and mechanisms of each source of error, which helps both quantifying and mitigating them. Same energy here. “Be more careful” is extraordinarily hand-wavy for a profession that calls itself engineering.

Aeolun 2y ago

> What I want from a post mortem is to know how we could prevent, detect or mitigate similar incidents in future and to make those changes to code or process. The answer this post gives to that bizarre question that always gets asked, is ‘nothing’, unless you want to significantly adjust the speed that we deliver features. Any added process or check is going to impose overhead and make the team a little bit less happy. Ocassionally you’ll have a unicorn situation where there is actually a relatively simple fix, but those are few and far between. In extremis, you’re reduced to a situation in which you have zero incidents, but you also have zero work getting done.

hamandcheese 2y ago
> Ocassionally you’ll have a unicorn situation where there is actually a relatively simple fix, but those are few and far between. Perhaps we have different backgrounds, but even in late stage startups I find there is an abundance of low hanging fruit and simple fixes. I'm sure it's different at Google, though.
scott_w 2y ago
That’s simply not true. Some processes are good, some post-mortem outcomes will focus on improving deployment speed (so you can revert changes faster), or improve your monitors so you can detect (and mitigate) incidents faster. On the other hand, enforcing a manual external QA check on every release WILL slow things down. You’re repeating the same mistake as the article by assuming “process” sits on a grade that naturally slows work down. This is because you’re not being precise in your reasoning. Look at the specifics and make a decision based on the facts in front of you.
- braza 2y ago
  > This is because you’re not being precise in your reasoning. Look at the specifics and make a decision based on the facts in front of you I agree with the premise here, but in my experience running incidents review the issue that I see it’s a mixture of a performatic safetycism with reactivity. Processes are the cheap bandaid to fix design, architectural and cultural issues. Most of the net positive micro-reforms that we had after incident reviews were the ones that invested in safety nets, faster recoveries, and guardrailing than a new process and will tax everyone.
  - scott_w 2y ago
    > Processes are the cheap bandaid to fix design, architectural and cultural issues. They can be, yes. I have a friend that thinks I'm totally insane by wanting to release code to production multiple times a day. His sweet spot is once every 2 weeks because he wants QA to check over every change. Most of his employers can manage once a month at best, and once a quarter is more typical. > Most of the net positive micro-reforms that we had after incident reviews were the ones that invested in safety nets, faster recoveries, and guardrailing than a new process and will tax everyone. I 100% agree with this. Your comment also reminded me to say that incident reviews are necessary but not sufficient. You also need engineering leadership reviewing at a higher-level to make bigger organisational or technical changes to further improve things.

shubhamjain 2y ago

Thanks for saying this. This mindset has infiltrated all the engineering teams and made software development hell for those of us who actually like shipping. Being more careful, adding more checks and processes, has exponentially less returns (just invert the graph). Somehow though, teams have been lead to believe that each incident needs to be responded with more processes.

nicbou 2y ago
This is an even bigger problem putside of our profession. Organisations do everything in their power to reduce the agency of employees and the general public through process. A company would rather spend 20 hours verifying a purchase than allow an unnecessary purchase every now and then. German culture in particular seems to favour this to an extreme.
- braza 2y ago
  I like the German workplace (to white collar job) for several reasons but this one is one that drives me away. We used to have an issue of deployments breaking in production, and one of the reasons was that we did not have some kind of smoke test in the post deployment (in our case we only had a rolling update as a strategy). The rational solution was only create that post-deployment step. The solution that our German managers demanded: cut access to deployment for the entire team, “on-call” to check the deployments, and a deployment spreadsheet to track it.
  - intelVISA 2y ago
    Good lord, are all those non-tech managers the reason Europe can't seem to build viable tech companies?
    - nicbou 2y ago
      In my experience it’s a cultural problem in Germany. Everything has to be done methodically, even if the method adds a disproportionate amount of friction. Often, the purported benefits are not even there. The thoroughness is full of holes, the diligence never done, the follow-ups never happening. It leads to situations where you need a certificate from your landlord that you take to the certified locksmith that your landlord contracted and show a piece of ID to order a key double that arrives 3 business days later at a cost of 60€. A smart German knows that there’s a locksmith in the basement of a nearby shopping mall that will gladly duplicate any key without a fuss, but even then the price is inflated by the authorised locksmiths. I document German bureaucracy for immigrants. Everything is like this. Every time I think “it can’t really be this ridiculous, I’m being uncharitable”, a colleague has a story that confirms that the truth is even more absurd. It’s funny until you realise the cost it has for society at large. All the wasted labour, all the bottlenecks, and little to show for it.

hamandcheese 2y ago

I get where you are coming from, and I certainly do expect actual facts, data, and reasoning to be a part of any serious postmortem analysis. But those will almost always be in relation to a very specific circumstance. I think there is still room for generalized parables such as this article - otherwise, we would be reading a postmortem blog post, which are also common here and usually do contain what you are asking for.

scott_w 2y ago
I think you can generalise without resorting to silly games like the article does. I gave some examples in a sibling comment that are high level enough to give an idea of the types of things I’d think about, without locking in to a specific incident I was part of.

odo1242 2y ago

Isn't that the point of the story though? To say "we need to have a conversation about facts, rather than something like always saying "we need to be more careful next time" when there's a problem? As far as I can tell, the author doesn't really give any generalized advice on how careful you should be, he's just pointing at the "carefulness dial" and telling people to make an informed decision.

scott_w 2y ago
I'm not really sure it is the author's point. I re-read the article to try and find your interpretation and I couldn't really find it there. Maybe a slight hint in the coda?

1oooqooq 2y ago

you're probably on the spectrum, as most anyone here. there's a tradeoff on shipping garbage fast which won't explode on your hands and getting promoted. and there's also the political art of selling what you want to other people by masking it as what they want. you and i and most people here will never understand any of that. good luck. people who do understand will have the careless knob stuck at 11. these analogies help us rational people point out the BS at least, without having to fight the BSer.

scott_w 2y ago
> you're probably on the spectrum, as most anyone here. Your comment starts with an ableist slur so I’m sure it’s going to be good /s > you and i and most people here will never understand any of that. good luck. people who do understand will have the careless knob stuck at 11. Nah, reading this comment wasn’t worthwhile after all. > these analogies help us rational people point out the BS at least, without having to fight the BSer. How cute, you think you’re “rational.”

astura 2y ago

Exactly, these stories only appeal to children and childish adults.

scott_w 2y ago
I think we can criticise the analogy as reductive without insulting people, here.
gffrd 2y ago
Or maybe put more accurately: these stories don’t appeal to you and people like you. What is your objection? Oversimplification? Lack of utility?

picometer 2y ago

Agreed. I recall being taught in college physics labs: there is no such thing as “human error”. Instead, think about the causes and mechanisms of each source of error, which helps both quantifying and mitigating them. Same energy here. “Be more careful” is extraordinarily hand-wavy for a profession that calls itself engineering.