4 ms·

“Oscar announces, “I’m seeing some log entries from the web server that look a little weird. I’m gonna look at those.” This is the beginning of a 5-minute silen

by cwillu 3y ago

“Oscar announces, “I’m seeing some log entries from the web server that look a little weird. I’m gonna look at those.” This is the beginning of a 5-minute silence. […] So it’s incumbent on you to interrupt this silence.” This is “we need to do something, this is something, we need to do it” thinking. The role of the commander imo is to insulate the investigators from exactly this sort of meaningless interruption. ““I need 5 minutes” [...] There is, however, a healthy kind of dead air.” If you need to be told this, you are being managed by your staff, not managing them.

BryantD 3y ago
The problem is that, as incident commander, I don’t know the difference between “I’m not saying anything because I’m stuck” and “I’m not saying anything because it’s going to take me five minutes.” The correct rule of thumb is to always over communicate, regardless of your role. If you’re troubleshooting, tell the incident commander that you’re doing X and it’ll take around 10 minutes before you see results. Then you’ve set expectations.
- bee_rider 3y ago
  It would be sort of unprofessional I guess, but it seems like it would actually help if people could sing mindless tunes as they work. If somebody is going “do-do-do, doot dadodo” as they work, you know they are in progress. A sudden “hmm” or silence indicates trouble.
  - tetha 3y ago
    But that's not over communication, that's just pollution of radio space. For example, in a recent incident I ended up kinda in charge / responsible for the database as the application encountered some weird lifelock with row and table locks. In such a situation, if no one needed the radio space, I ended up announcing the status of the database every few minutes, even if nothing really changed. Or, if someone does something, give feedback if this has a noticeable effect on your system or not. Or quickly ask if you want to do something that could affect them, no matter how little that is. "App, I'll modify setting xyz, tell me if that has an effect".
    - mlrtime 3y ago
      >I ended up announcing the status of the database every few minutes, even if nothing really changed. Yes exactly, in a incident if there is some long task that is running that will clear the incident, this is important. It's essentially a progress bar.
  - Chris2048 3y ago
    aka The Everything's Okay Alarm
- marginalia_nu 3y ago
  That's the sort of anxious micromanagement a good manager is shielding their team from in a situation like this. You need to trust your guys. They're the experts.
  - mlrtime 3y ago
    This is all in the context of a incident and the role of a IC. The author nails all the issues here. Dead silence with no shared screen is not good for a incident (without any other context). You either communicate constantly on what is happening or say "I need 10 minutes to debug" In which case 10 minutes is given but they must come back with an update at that time. an IC absolutely cannot just let someone go off and trust they are debugging the issue.
    - Chris2048 3y ago
      Why? To any of this? If you aren't fixing/investigating the problem, why get in the way of the people who are on the basis of trust issues, or some ICs idea of what's "good for a incident"?
      - mlrtime 3y ago
        How do you know you are getting in the way, how do you know the guy just didn't go out to lunch? How long do you wait to hear back, do you just wait forever? The scenario I have in my head is a conference, the person in the example has no video, no shared screen and no response. The 5 minutes is just an example, maybe a bad one but the idea still holds.
        Chris2048 3y ago
        > how do you know the guy just didn't go out to lunch? You don't, you hire people you can trust, and with whom what is/isn't appropriate during an incident is clear. > How long do you wait to hear back, do you just wait forever? A short message "How's that going?" if it seems to be taking a long time and there is no communication. What you are describing is micromanaging the actions of the people who are meant to be experts in that domain. Why is this needed? What issues have arisen before, and do these issues arise during normal development. > person in the example It's not clear to me the role the people in the example have. What does being "primary investigator" mean? How does that relate to Ops - why isn't support, or the devs investigating the logs. Who in the example would be knowledgeable about this area? Is the Ops person qualified to know if a web log is strange? Why would they be pushing fixes to a web server? To me, if people in the know are competent enough to investigate an issue, they should be competent enough to communicate about it.
wildrhythms 3y ago
I read that part as meaning this probably doesn't need to be a call at all at this point in the investigating process.
sanderjd 3y ago
Yeah this article didn't go where I expected it to go. Silence is nearly always an indication that someone is doing something or thinking about something, and the advice here is to interrupt them?
- mlrtime 3y ago
  But how do you know that without any other information? In a high stakes incident you cannot let radio silence go off without clear communication on status updates... this was called out in the bottom.
  - sanderjd 3y ago
    You train people, delegate to them, and then trust them to do the right thing. This is discussed in the article as well, but I think expectations are much better than interruptions. And even then, it is better to set those expectations during training rather than during the response. People should be trained to prioritize communication during a response, but not above their work doing the response. And the people doing the work are individually best suited to make the call of whether what they're doing is more or less important than communicating. In my view, it's really hard to overstate how hard it is to be reading big volumes of logs, reconstructing the runtime state of some big complex distributed system from the breadcrumbs available, and thinking about what to do next to most quickly mitigate or get more information on the issue, while being bombarded with messages.

by cwillu 3y ago

BryantD 3y ago

The problem is that, as incident commander, I don’t know the difference between “I’m not saying anything because I’m stuck” and “I’m not saying anything because it’s going to take me five minutes.” The correct rule of thumb is to always over communicate, regardless of your role. If you’re troubleshooting, tell the incident commander that you’re doing X and it’ll take around 10 minutes before you see results. Then you’ve set expectations.

bee_rider 3y ago
It would be sort of unprofessional I guess, but it seems like it would actually help if people could sing mindless tunes as they work. If somebody is going “do-do-do, doot dadodo” as they work, you know they are in progress. A sudden “hmm” or silence indicates trouble.
- tetha 3y ago
  But that's not over communication, that's just pollution of radio space. For example, in a recent incident I ended up kinda in charge / responsible for the database as the application encountered some weird lifelock with row and table locks. In such a situation, if no one needed the radio space, I ended up announcing the status of the database every few minutes, even if nothing really changed. Or, if someone does something, give feedback if this has a noticeable effect on your system or not. Or quickly ask if you want to do something that could affect them, no matter how little that is. "App, I'll modify setting xyz, tell me if that has an effect".
  - mlrtime 3y ago
    >I ended up announcing the status of the database every few minutes, even if nothing really changed. Yes exactly, in a incident if there is some long task that is running that will clear the incident, this is important. It's essentially a progress bar.
- Chris2048 3y ago
  aka The Everything's Okay Alarm
marginalia_nu 3y ago
That's the sort of anxious micromanagement a good manager is shielding their team from in a situation like this. You need to trust your guys. They're the experts.
- mlrtime 3y ago
  This is all in the context of a incident and the role of a IC. The author nails all the issues here. Dead silence with no shared screen is not good for a incident (without any other context). You either communicate constantly on what is happening or say "I need 10 minutes to debug" In which case 10 minutes is given but they must come back with an update at that time. an IC absolutely cannot just let someone go off and trust they are debugging the issue.
  - Chris2048 3y ago
    Why? To any of this? If you aren't fixing/investigating the problem, why get in the way of the people who are on the basis of trust issues, or some ICs idea of what's "good for a incident"?
    - mlrtime 3y ago
      How do you know you are getting in the way, how do you know the guy just didn't go out to lunch? How long do you wait to hear back, do you just wait forever? The scenario I have in my head is a conference, the person in the example has no video, no shared screen and no response. The 5 minutes is just an example, maybe a bad one but the idea still holds.
      - Chris2048 3y ago
        > how do you know the guy just didn't go out to lunch? You don't, you hire people you can trust, and with whom what is/isn't appropriate during an incident is clear. > How long do you wait to hear back, do you just wait forever? A short message "How's that going?" if it seems to be taking a long time and there is no communication. What you are describing is micromanaging the actions of the people who are meant to be experts in that domain. Why is this needed? What issues have arisen before, and do these issues arise during normal development. > person in the example It's not clear to me the role the people in the example have. What does being "primary investigator" mean? How does that relate to Ops - why isn't support, or the devs investigating the logs. Who in the example would be knowledgeable about this area? Is the Ops person qualified to know if a web log is strange? Why would they be pushing fixes to a web server? To me, if people in the know are competent enough to investigate an issue, they should be competent enough to communicate about it.

wildrhythms 3y ago

I read that part as meaning this probably doesn't need to be a call at all at this point in the investigating process.

sanderjd 3y ago

Yeah this article didn't go where I expected it to go. Silence is nearly always an indication that someone is doing something or thinking about something, and the advice here is to interrupt them?

mlrtime 3y ago
But how do you know that without any other information? In a high stakes incident you cannot let radio silence go off without clear communication on status updates... this was called out in the bottom.
- sanderjd 3y ago
  You train people, delegate to them, and then trust them to do the right thing. This is discussed in the article as well, but I think expectations are much better than interruptions. And even then, it is better to set those expectations during training rather than during the response. People should be trained to prioritize communication during a response, but not above their work doing the response. And the people doing the work are individually best suited to make the call of whether what they're doing is more or less important than communicating. In my view, it's really hard to overstate how hard it is to be reading big volumes of logs, reconstructing the runtime state of some big complex distributed system from the breadcrumbs available, and thinking about what to do next to most quickly mitigate or get more information on the issue, while being bombarded with messages.