3 ms·

That's the sort of anxious micromanagement a good manager is shielding their team from in a situation like this. You need to trust your guys. They're the expe

That's the sort of anxious micromanagement a good manager is shielding their team from in a situation like this. You need to trust your guys. They're the experts.

mlrtime 3y ago
This is all in the context of a incident and the role of a IC. The author nails all the issues here. Dead silence with no shared screen is not good for a incident (without any other context). You either communicate constantly on what is happening or say "I need 10 minutes to debug" In which case 10 minutes is given but they must come back with an update at that time. an IC absolutely cannot just let someone go off and trust they are debugging the issue.
- Chris2048 3y ago
  Why? To any of this? If you aren't fixing/investigating the problem, why get in the way of the people who are on the basis of trust issues, or some ICs idea of what's "good for a incident"?
  - mlrtime 3y ago
    How do you know you are getting in the way, how do you know the guy just didn't go out to lunch? How long do you wait to hear back, do you just wait forever? The scenario I have in my head is a conference, the person in the example has no video, no shared screen and no response. The 5 minutes is just an example, maybe a bad one but the idea still holds.
    - Chris2048 3y ago
      > how do you know the guy just didn't go out to lunch? You don't, you hire people you can trust, and with whom what is/isn't appropriate during an incident is clear. > How long do you wait to hear back, do you just wait forever? A short message "How's that going?" if it seems to be taking a long time and there is no communication. What you are describing is micromanaging the actions of the people who are meant to be experts in that domain. Why is this needed? What issues have arisen before, and do these issues arise during normal development. > person in the example It's not clear to me the role the people in the example have. What does being "primary investigator" mean? How does that relate to Ops - why isn't support, or the devs investigating the logs. Who in the example would be knowledgeable about this area? Is the Ops person qualified to know if a web log is strange? Why would they be pushing fixes to a web server? To me, if people in the know are competent enough to investigate an issue, they should be competent enough to communicate about it.
andrewaylett 3y ago
The incident commander isn't necessarily a manager. Indeed, where I work it would be uncommon for a manager to take incident command and unlikely that anyone above the first level of management would join the incident call.
BryantD 3y ago
Not at all. The way you gain trust is by making clear commitments and either meeting them or updating them as necessary. I'm not saying that the engineering troubleshooting a problem should be narrating moment by moment. I'm saying they should say, at the outset, something like: "I'm going to go try flushing the cache and restarting the app servers. I expect this to take about fifteen minutes, and if it works I would expect to see the database load normalize." That's all I need as an incident manager. Otherwise I'm sitting there not knowing what to say in my status updates, my stakeholders are asking questions about estimated duration that I can't answer, and so on.
- tetha 3y ago
  And as an example, since we've been dealing with a huge amount of incidents over the last few weeks: To me, it's completely normal if BryantD is like "Yo tetha, how are the caches and restarts coming?" and it doesn't take much for me to mumble something like "16 out of 28 done. database, you see anything changing?" No need to be vastly eloquent or anything. Or I might throw something like that in on my own if things are dead silent right now and no one else needs the radio. And I think that's a good habit other people on the team are picking up. If someone needs the air for something important, they can talk, but otherwise you have various status updates floating around. And more often than not, those result in "Wait, did you just say xyz changed? Did I just do something useful?"
  - BryantD 3y ago
    I see we'd enjoy working together. ;) Out of curiosity, how do you parallelize incident response? I am fairly picky about only changing one thing at once but I'm not sure I'm always right about that.
    - tetha 3y ago
      To me, this depends on the different failure domains and how they interact with each other and based off of that, it can be decided if the teams should coordinate changes, communicate changes or just run. For example we've had situations in which... creative user and amazing code managed to fry both the dedicated elasticsearch cluster for the application as well as the dedicated database, and the application servers were also weird. Here, we'd split into three small teams each responsible for each of these pieces. And looking at these failure domains - ES and postgres don't interact with each other, so both of these teams should just run independently as fast as they can to get their components working again and inform the app team when they are back up. However, the application team should closely coordinate their actions with both of these teams - I've had enough situations in which someone pressed the "hilarious load on ES" button while people were still getting ES up to capacity... and down we go again. Or in a similar way, we had a central database outage and a couple dozen apps got taken out. Database in the lead, sure, but once that's going again, the different application teams can run free and make changes with loose coordination with the database team. However, within the same failure domain, I really don't like to make too many changes in parallel. Getting back up ASAP is a priority, for sure, but what about the outage in 2 hours, or tomorrow because we just hit the system with a couple of wrenches and have no idea why it's back up? Here I strongly prefer deliberate, individual changes so we can get an idea why the system failed. 10 more minutes of downtime / degraded service now can safe us many nerves and downtime hours over the next few days.

That's the sort of anxious micromanagement a good manager is shielding their team from in a situation like this. You need to trust your guys. They're the experts.

mlrtime 3y ago

This is all in the context of a incident and the role of a IC. The author nails all the issues here. Dead silence with no shared screen is not good for a incident (without any other context). You either communicate constantly on what is happening or say "I need 10 minutes to debug" In which case 10 minutes is given but they must come back with an update at that time. an IC absolutely cannot just let someone go off and trust they are debugging the issue.

Chris2048 3y ago
Why? To any of this? If you aren't fixing/investigating the problem, why get in the way of the people who are on the basis of trust issues, or some ICs idea of what's "good for a incident"?
- mlrtime 3y ago
  How do you know you are getting in the way, how do you know the guy just didn't go out to lunch? How long do you wait to hear back, do you just wait forever? The scenario I have in my head is a conference, the person in the example has no video, no shared screen and no response. The 5 minutes is just an example, maybe a bad one but the idea still holds.
  - Chris2048 3y ago
    > how do you know the guy just didn't go out to lunch? You don't, you hire people you can trust, and with whom what is/isn't appropriate during an incident is clear. > How long do you wait to hear back, do you just wait forever? A short message "How's that going?" if it seems to be taking a long time and there is no communication. What you are describing is micromanaging the actions of the people who are meant to be experts in that domain. Why is this needed? What issues have arisen before, and do these issues arise during normal development. > person in the example It's not clear to me the role the people in the example have. What does being "primary investigator" mean? How does that relate to Ops - why isn't support, or the devs investigating the logs. Who in the example would be knowledgeable about this area? Is the Ops person qualified to know if a web log is strange? Why would they be pushing fixes to a web server? To me, if people in the know are competent enough to investigate an issue, they should be competent enough to communicate about it.

andrewaylett 3y ago

The incident commander isn't necessarily a manager. Indeed, where I work it would be uncommon for a manager to take incident command and unlikely that anyone above the first level of management would join the incident call.

BryantD 3y ago

Not at all. The way you gain trust is by making clear commitments and either meeting them or updating them as necessary. I'm not saying that the engineering troubleshooting a problem should be narrating moment by moment. I'm saying they should say, at the outset, something like: "I'm going to go try flushing the cache and restarting the app servers. I expect this to take about fifteen minutes, and if it works I would expect to see the database load normalize." That's all I need as an incident manager. Otherwise I'm sitting there not knowing what to say in my status updates, my stakeholders are asking questions about estimated duration that I can't answer, and so on.

tetha 3y ago
And as an example, since we've been dealing with a huge amount of incidents over the last few weeks: To me, it's completely normal if BryantD is like "Yo tetha, how are the caches and restarts coming?" and it doesn't take much for me to mumble something like "16 out of 28 done. database, you see anything changing?" No need to be vastly eloquent or anything. Or I might throw something like that in on my own if things are dead silent right now and no one else needs the radio. And I think that's a good habit other people on the team are picking up. If someone needs the air for something important, they can talk, but otherwise you have various status updates floating around. And more often than not, those result in "Wait, did you just say xyz changed? Did I just do something useful?"
- BryantD 3y ago
  I see we'd enjoy working together. ;) Out of curiosity, how do you parallelize incident response? I am fairly picky about only changing one thing at once but I'm not sure I'm always right about that.
  - tetha 3y ago
    To me, this depends on the different failure domains and how they interact with each other and based off of that, it can be decided if the teams should coordinate changes, communicate changes or just run. For example we've had situations in which... creative user and amazing code managed to fry both the dedicated elasticsearch cluster for the application as well as the dedicated database, and the application servers were also weird. Here, we'd split into three small teams each responsible for each of these pieces. And looking at these failure domains - ES and postgres don't interact with each other, so both of these teams should just run independently as fast as they can to get their components working again and inform the app team when they are back up. However, the application team should closely coordinate their actions with both of these teams - I've had enough situations in which someone pressed the "hilarious load on ES" button while people were still getting ES up to capacity... and down we go again. Or in a similar way, we had a central database outage and a couple dozen apps got taken out. Database in the lead, sure, but once that's going again, the different application teams can run free and make changes with loose coordination with the database team. However, within the same failure domain, I really don't like to make too many changes in parallel. Getting back up ASAP is a priority, for sure, but what about the outage in 2 hours, or tomorrow because we just hit the system with a couple of wrenches and have no idea why it's back up? Here I strongly prefer deliberate, individual changes so we can get an idea why the system failed. 10 more minutes of downtime / degraded service now can safe us many nerves and downtime hours over the next few days.