This article was really great, and this gem stood out to me: > When the hypothesis failed, we again rushed to move on with another work. We didn’t take the tim

This article was really great, and this gem stood out to me: > When the hypothesis failed, we again rushed to move on with another work. We didn’t take the time to analyze why our theory was wrong. We didn’t question the experiment’s validity, which would have led us to realize our patch was wrong early on. I've read dozens of postmortems where "someone did something to 'prove' that X wasn't the cause, but after ruling everything else out the team eventually realised that X was in fact the cause.", with 'X' being everything from "this patch can't have caused that behavior" to "I know it's not the kernel update because how could that have changed anything?" In the face of a crisis, if you are in charge you really have to take all the "facts" you are presented with as being contingent on the level of panic that the presenter is feeling -- being a great engineer is more or less orthogonal to calm under pressure.