4 ms·
These sort of deeply apologetic and hyper-transparent post-mortems have become commonplace, but sometimes I wonder how beneficial they are. Customers appreciat
by katzgrau 10y ago
These sort of deeply apologetic and hyper-transparent post-mortems have become commonplace, but sometimes I wonder how beneficial they are.
Customers appreciate transparency, but perhaps delving into the fine details of the investigation (various hypotheses, overlooked warning signs, yada yada) might actually end up leaving the customer more unsettled than they would have been otherwise.
Today I learned that Asana had a bunch of bad deploys and put the icing on the cake with one that resulted in an outage the next day.
This is coming from someone who runs an ad server - if that ad server goes down it's damn near catastrophic for my customers and their customers. When we do have a (rare) outage, I sweat it out, reassure customers that people are on it, and give a brief, accurate, and high level explanation without getting into the gruesome details.
I'm not saying my approach is best, but I do think trying to avoid scaring people in your explanation is an idea.
- bognition 10y agoI work at a shop that does these kinds of post mortems. I find them highly beneficial. They require us to actually do the work of identifying the issues and writing up what happened and why. I realize that having a customer contract to do this shouldn't be a requirement but human psychology is funny thing. I can turn to my pm and say "I have to do this it's part of the contract" and they immediately back off. I agree it might not be the best solution but it's definitely better than not doing them.
- dogma1138 10y agoI think the OP didn't mean that these post mortems are not beneficial internally, what he said that disclosing all these details to the public can be confusing and maybe counter productive.
- merb 10y agoI'm not sure whats better. 1. Describing the root cause and what you failed at. 2. Blame the stuff you are using / other people (clouds you use) 3. just says nothing and try to forget what happened. What do you think is best?
- zzzcpan 10y agoThe post like this is definitely confusing. I think you do need to at least acknowledge the problem. With a clear non-technical explanation of the problem in the first paragraph. The rest should go into real technical details of the result of the investigation, not an investigation itself.
- shiven 10y agoWhy is it confusing? I did not find it confusing. I like such excruciatingly detailed postmortem analyses, they make for great reading and my respect for the company that does this is increased by reading these.
- themartorana 10y agoIt's maybe a level of details thing. "A bad deploy went unnoticed, causing a cascading failure. We identified how that happened and have new checks in place to prevent it in the future." Two lines, with the same information someone not very technically literate would understand from the OP. I agree with being transparent, but I also believe in not unnecessarily scaring and/or confusing customers, either. (Pretty soon they'll just start outting individual engineers...)
- marcog1 10y agoWe will never out individuals. The person who committed the code was innocent. We got him a fun gift as a sort of joke.
- merb 10y agoYep the best thing some could do: - train the people more - help them to get over (some ppl could be really mad and infconfident after they did bad)
- themartorana 10y agoThat was entirely tongue-in-cheek. I wouldn't ever expect you to do that! It was an exaggerated example.
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- marcog1 10y agoEvery response from our users so far has been thanking is for the transparency. It also represents our internal transparency, and that has a real impact on recruiting.
- noir_lord 10y agoNot an Asana user but if I were this kind of response is exactly what I like to see as a both a user and a developer so well done.
- rossjudson 10y agoYour approach works for the incident, but not for the relationship. Transparency about the technical nature of the outage is a commitment to the client that this type of outage won't recur, and steps are being taken to ensure that. It pierces the veil of arrogance by assuming client competence. That client is actually someone who reports to someone else, and they're going to have to explain their outage to the boss. For cloud providers, this kind of transparent post-mortem is the root of a fan-out of incident analysis.
- smoe 10y agoPersonally I'd produce both. A brief high level explanation for non technical people (e.G. customers, press) and an in-depth blog post with the gruesome details. The latter is useful for example when my boss asks me to evaluate whether to continue using a service after an incident. If I can't get enough information to make a recommendation I might propose a switch out of distrust. Especially when to problem was related to security or privacy.