3 ms·

An old colleague of mine worked pretty extensively with a backend engineer on a script to do some data migration on user accounts. The other engineer did most o

by geuis 4y ago

An old colleague of mine worked pretty extensively with a backend engineer on a script to do some data migration on user accounts. The other engineer did most of the work on the script itself. Mind you, the company has millions of users and didn't use staging databases. (No idea if this is still true.) Come D-day, my colleague runs the script on a limited group of users (a 100k or so) to validate. I forget the details, but something in the script was incorrect and ended up breaking some features for all of those users. Once reports started coming in, they were super worried and semi-freaked out. A war room was setup that day and all the people involved jumped in. One of the first things that happened after determining what happened was to calm them down and reassure they weren't in trouble. After that the group worked on a solution for a few hours and established a plan to fix everything. I was actually surprised that the response was so well handled. There was no finger pointing and just a group effort to fix the problem. To me, that's how every problem should be handled, and not instilling fear for losing your job if something bad potentially happens. To anyone who wants to leave replies about staging databases, bad dev practices, etc, don't bother please. This was years ago and it was how things were done at the company. Our team was not part of the backend team or infra and worked with lots of areas of engineering on different issues.

schoolornot 4y ago
Which is precisely why I don't understand the purpose to even have postmortems for 95% of outages. If everyone is aware of what went wrong and the issue is unlikely to ever happen again, what is the point? Well, at companies of the size I work with it is to point fingers, make PMs feel more important, and give people talking points.
- joshuamorton 4y ago
  A postmortem is, in most cases, really just a formal process to ensure everyone (including leadership) knows what went wrong, and how to prevent the issue from happening again, with some associated ceremony about like actually keeping track of the relevant action items to prevent recurrence of the issue. It's pretty rare that you run into an issue that can't reoccur, or have similarly shaped reoccurrences.
- s1gnp0st 4y ago
  If you're having enough outages that postmortems are burdensome, I don't think the problem is the number of postmortems.
- adamckay 4y ago
  > If everyone is aware of what went wrong and the issue is unlikely to ever happen again, what is the point? Because the only way you can make everyone aware is to write it down. Anything else is hearsay. And going through the process of a thorough postmortem can ensure you do know exactly what went wrong and why, and how you can prevent the same and similar issues from happening again in the future. Perhaps from this example it serves as documented proof that work on setting up staging databases needs to be prioritised and invested in? Maybe it's that scripts such as this should be reviewed by another engineer before running? Maybe the standard operating procedure is updated so a backup is taken immediately before running any scripts that write to the database? Maybe you create a rule to limit the blast radius in future and do smaller roll outs to 1k users instead of 100k? Maybe scripts should be developed with a dry-run feature?
- MaulingMonkey 4y ago
  > If everyone is aware of what went wrong Big assumption. Chances are at least someone was out on vacation or something. A postmortem can help spread the word - and perhaps calm those downstream who weren't in the loop as much, through a show of clear and open communication and ownership, instead of trying to bury it. > and the issue is unlikely to ever happen again, Also a big assumption. You can almost certainly https://en.wikipedia.org/wiki/Five_whys https://en.wikipedia.org/wiki/Five_whys your way into a broader pattern of something that will happen again, and can review what worked to expectations, what didn't, what should be improved, and what can be left alone - even if it perhaps will happen again - perhaps being more expensive to try and fix, than to merely accept the occasional failure as an acceptable price of doing business.
- dasil003 4y ago
  > everyone is aware of what went wrong This is never true past a certain company size threshold, and even for smaller companies once you start asking the "five whys" you see there is never just one root cause. Even for straightforward cases where everyone generally agrees, it can still be informative to capture the analysis for future reference and review of reliability patterns. Incidents are a learning opportunity. If people are pointing fingers then that's a sign of a bigger cultural issue, one which will not be solved by avoiding discussing and documenting incidents.
shoo 4y ago
This is a fair anecdote about response an active production incident. While there is a live incident, the focus should be on resolving it, and restoring service to customers. But, this anecdote does not comment about a postmortem or other organizational habits that can be used to review historical incidents or near misses (after the incidents have been resolved and everyone has had time to rest), diagnose risks, identify contributing direct or indirect causes, then use this feedback and learning to set goals and drive changes to reduce the risk of similar incidents recurring in future.
- geuis 4y ago
  While I can't add too much to what I already said, I have heard semi-recently the company is working on dev or staging databases finally. Can't say the anecdote in question contributed to that, but it's nice to consider.
zulban 4y ago
Interesting anecdote but this is not about postmortems whatsoever.
- geuis 4y ago
  Colleague wrote a very detailed postmortem document detailing the issues that led to the issue. Also included details about impact and estimated time to fix. No one got fired, the issue was handled professionally, accounts were fixed, and the original ticket was eventually resolved to everyone's satisfaction. Sorry if I didn't capture certain details in my original post. I kind of just assumed most people reading this have been through the "everything is on fire" experience and would get that the postmortem was resolved successfully. Not everything has to end in a hanging or massive change to an organization.

by geuis 4y ago

schoolornot 4y ago

Which is precisely why I don't understand the purpose to even have postmortems for 95% of outages. If everyone is aware of what went wrong and the issue is unlikely to ever happen again, what is the point? Well, at companies of the size I work with it is to point fingers, make PMs feel more important, and give people talking points.

joshuamorton 4y ago
A postmortem is, in most cases, really just a formal process to ensure everyone (including leadership) knows what went wrong, and how to prevent the issue from happening again, with some associated ceremony about like actually keeping track of the relevant action items to prevent recurrence of the issue. It's pretty rare that you run into an issue that can't reoccur, or have similarly shaped reoccurrences.
s1gnp0st 4y ago
If you're having enough outages that postmortems are burdensome, I don't think the problem is the number of postmortems.
adamckay 4y ago
> If everyone is aware of what went wrong and the issue is unlikely to ever happen again, what is the point? Because the only way you can make everyone aware is to write it down. Anything else is hearsay. And going through the process of a thorough postmortem can ensure you do know exactly what went wrong and why, and how you can prevent the same and similar issues from happening again in the future. Perhaps from this example it serves as documented proof that work on setting up staging databases needs to be prioritised and invested in? Maybe it's that scripts such as this should be reviewed by another engineer before running? Maybe the standard operating procedure is updated so a backup is taken immediately before running any scripts that write to the database? Maybe you create a rule to limit the blast radius in future and do smaller roll outs to 1k users instead of 100k? Maybe scripts should be developed with a dry-run feature?
MaulingMonkey 4y ago
> If everyone is aware of what went wrong Big assumption. Chances are at least someone was out on vacation or something. A postmortem can help spread the word - and perhaps calm those downstream who weren't in the loop as much, through a show of clear and open communication and ownership, instead of trying to bury it. > and the issue is unlikely to ever happen again, Also a big assumption. You can almost certainly https://en.wikipedia.org/wiki/Five_whys https://en.wikipedia.org/wiki/Five_whys your way into a broader pattern of something that will happen again, and can review what worked to expectations, what didn't, what should be improved, and what can be left alone - even if it perhaps will happen again - perhaps being more expensive to try and fix, than to merely accept the occasional failure as an acceptable price of doing business.
dasil003 4y ago
> everyone is aware of what went wrong This is never true past a certain company size threshold, and even for smaller companies once you start asking the "five whys" you see there is never just one root cause. Even for straightforward cases where everyone generally agrees, it can still be informative to capture the analysis for future reference and review of reliability patterns. Incidents are a learning opportunity. If people are pointing fingers then that's a sign of a bigger cultural issue, one which will not be solved by avoiding discussing and documenting incidents.

shoo 4y ago

This is a fair anecdote about response an active production incident. While there is a live incident, the focus should be on resolving it, and restoring service to customers. But, this anecdote does not comment about a postmortem or other organizational habits that can be used to review historical incidents or near misses (after the incidents have been resolved and everyone has had time to rest), diagnose risks, identify contributing direct or indirect causes, then use this feedback and learning to set goals and drive changes to reduce the risk of similar incidents recurring in future.

geuis 4y ago
While I can't add too much to what I already said, I have heard semi-recently the company is working on dev or staging databases finally. Can't say the anecdote in question contributed to that, but it's nice to consider.

zulban 4y ago

Interesting anecdote but this is not about postmortems whatsoever.

geuis 4y ago
Colleague wrote a very detailed postmortem document detailing the issues that led to the issue. Also included details about impact and estimated time to fix. No one got fired, the issue was handled professionally, accounts were fixed, and the original ticket was eventually resolved to everyone's satisfaction. Sorry if I didn't capture certain details in my original post. I kind of just assumed most people reading this have been through the "everything is on fire" experience and would get that the postmortem was resolved successfully. Not everything has to end in a hanging or massive change to an organization.