3 ms·
A great story which really resonated with me from the operations / reliability engineering perspective. It is very common for junior engineers to jump into an a
by falcor84 8y ago
A great story which really resonated with me from the operations / reliability engineering perspective. It is very common for junior engineers to jump into an action in case of a big outage before really trying to
understand the problem.
One example I've seen multiple times was to deal with a repeatedly crashing process by restarting yet another process. In some cases, this may help, but in others it can make the situation much much worse. Instead, the advice I took to heart is to delay action until you have a mental model of the problem, and how the planned action would resolve it (and if possible, sense check it with another engineer).
In the best case scenario, you would have a trusted (well-tested) mitigation procedure that would provide you with enough time to recreate the issue and test the proposed solution in a staging environment.
- gugagore 8y agoI think when something is very uncertain (because it's hard and/or the engineer is inexperienced), it's not possible to develop a mental model that lets you plan a course of action. However, taking a slightly directed, non-destructive action can sometimes reveal more information to you. The idealization of this is coming up with some hypothesis, and testing it, which I think is covered by your approach. But what if you don't know enough to even know the right questions to ask. You may be dead in the water if you don't allow an unplanned action.
- falcor84 8y agoAgreed, there definitely no magic bullet hate. At the end of the day it's all about being pragmatic.