6 ms·

The problem is that, as incident commander, I don’t know the difference between “I’m not saying anything because I’m stuck” and “I’m not saying anything because

The problem is that, as incident commander, I don’t know the difference between “I’m not saying anything because I’m stuck” and “I’m not saying anything because it’s going to take me five minutes.” The correct rule of thumb is to always over communicate, regardless of your role. If you’re troubleshooting, tell the incident commander that you’re doing X and it’ll take around 10 minutes before you see results. Then you’ve set expectations.

bee_rider 3y ago
It would be sort of unprofessional I guess, but it seems like it would actually help if people could sing mindless tunes as they work. If somebody is going “do-do-do, doot dadodo” as they work, you know they are in progress. A sudden “hmm” or silence indicates trouble.
- tetha 3y ago
  But that's not over communication, that's just pollution of radio space. For example, in a recent incident I ended up kinda in charge / responsible for the database as the application encountered some weird lifelock with row and table locks. In such a situation, if no one needed the radio space, I ended up announcing the status of the database every few minutes, even if nothing really changed. Or, if someone does something, give feedback if this has a noticeable effect on your system or not. Or quickly ask if you want to do something that could affect them, no matter how little that is. "App, I'll modify setting xyz, tell me if that has an effect".
  - mlrtime 3y ago
    >I ended up announcing the status of the database every few minutes, even if nothing really changed. Yes exactly, in a incident if there is some long task that is running that will clear the incident, this is important. It's essentially a progress bar.
- Chris2048 3y ago
  aka The Everything's Okay Alarm
marginalia_nu 3y ago
That's the sort of anxious micromanagement a good manager is shielding their team from in a situation like this. You need to trust your guys. They're the experts.
- mlrtime 3y ago
  This is all in the context of a incident and the role of a IC. The author nails all the issues here. Dead silence with no shared screen is not good for a incident (without any other context). You either communicate constantly on what is happening or say "I need 10 minutes to debug" In which case 10 minutes is given but they must come back with an update at that time. an IC absolutely cannot just let someone go off and trust they are debugging the issue.
  - Chris2048 3y ago
    Why? To any of this? If you aren't fixing/investigating the problem, why get in the way of the people who are on the basis of trust issues, or some ICs idea of what's "good for a incident"?
    - mlrtime 3y ago
      How do you know you are getting in the way, how do you know the guy just didn't go out to lunch? How long do you wait to hear back, do you just wait forever? The scenario I have in my head is a conference, the person in the example has no video, no shared screen and no response. The 5 minutes is just an example, maybe a bad one but the idea still holds.
      - Chris2048 3y ago
        > how do you know the guy just didn't go out to lunch? You don't, you hire people you can trust, and with whom what is/isn't appropriate during an incident is clear. > How long do you wait to hear back, do you just wait forever? A short message "How's that going?" if it seems to be taking a long time and there is no communication. What you are describing is micromanaging the actions of the people who are meant to be experts in that domain. Why is this needed? What issues have arisen before, and do these issues arise during normal development. > person in the example It's not clear to me the role the people in the example have. What does being "primary investigator" mean? How does that relate to Ops - why isn't support, or the devs investigating the logs. Who in the example would be knowledgeable about this area? Is the Ops person qualified to know if a web log is strange? Why would they be pushing fixes to a web server? To me, if people in the know are competent enough to investigate an issue, they should be competent enough to communicate about it.

bee_rider 3y ago

It would be sort of unprofessional I guess, but it seems like it would actually help if people could sing mindless tunes as they work. If somebody is going “do-do-do, doot dadodo” as they work, you know they are in progress. A sudden “hmm” or silence indicates trouble.

tetha 3y ago
But that's not over communication, that's just pollution of radio space. For example, in a recent incident I ended up kinda in charge / responsible for the database as the application encountered some weird lifelock with row and table locks. In such a situation, if no one needed the radio space, I ended up announcing the status of the database every few minutes, even if nothing really changed. Or, if someone does something, give feedback if this has a noticeable effect on your system or not. Or quickly ask if you want to do something that could affect them, no matter how little that is. "App, I'll modify setting xyz, tell me if that has an effect".
- mlrtime 3y ago
  >I ended up announcing the status of the database every few minutes, even if nothing really changed. Yes exactly, in a incident if there is some long task that is running that will clear the incident, this is important. It's essentially a progress bar.
Chris2048 3y ago
aka The Everything's Okay Alarm

marginalia_nu 3y ago

That's the sort of anxious micromanagement a good manager is shielding their team from in a situation like this. You need to trust your guys. They're the experts.

mlrtime 3y ago
This is all in the context of a incident and the role of a IC. The author nails all the issues here. Dead silence with no shared screen is not good for a incident (without any other context). You either communicate constantly on what is happening or say "I need 10 minutes to debug" In which case 10 minutes is given but they must come back with an update at that time. an IC absolutely cannot just let someone go off and trust they are debugging the issue.
- Chris2048 3y ago
  Why? To any of this? If you aren't fixing/investigating the problem, why get in the way of the people who are on the basis of trust issues, or some ICs idea of what's "good for a incident"?
  - mlrtime 3y ago
    How do you know you are getting in the way, how do you know the guy just didn't go out to lunch? How long do you wait to hear back, do you just wait forever? The scenario I have in my head is a conference, the person in the example has no video, no shared screen and no response. The 5 minutes is just an example, maybe a bad one but the idea still holds.
    - Chris2048 3y ago
      > how do you know the guy just didn't go out to lunch? You don't, you hire people you can trust, and with whom what is/isn't appropriate during an incident is clear. > How long do you wait to hear back, do you just wait forever? A short message "How's that going?" if it seems to be taking a long time and there is no communication. What you are describing is micromanaging the actions of the people who are meant to be experts in that domain. Why is this needed? What issues have arisen before, and do these issues arise during normal development. > person in the example It's not clear to me the role the people in the example have. What does being "primary investigator" mean? How does that relate to Ops - why isn't support, or the devs investigating the logs. Who in the example would be knowledgeable about this area? Is the Ops person qualified to know if a web log is strange? Why would they be pushing fixes to a web server? To me, if people in the know are competent enough to investigate an issue, they should be competent enough to communicate about it.