5 ms·

It's not inspiring to me with the 'cool tactical' framing this article attempts to convey. I've worked as an oncall for a fundamental backbone service of the i

It's not inspiring to me with the 'cool tactical' framing this article attempts to convey. I've worked as an oncall for a fundamental backbone service of the internet in the past and paged into middle of the night outages. It's harrowing and exhausting. Cool names like 'incident commander' do not change this. We also had a "see ya in the morning" culture. Instead I'd be much more impressed to have a "see ya in the afternoon, get some sleep" culture.

choeger 5y ago
It seems to be a bit of cargo cult, to be honest. They seem to take inspiration from ER teams or the military. I think that this kind of drill helps a lot for cases where you can take a pre-planned route, like deploying that backup server or rerouting traffic. But the obvious question then is: Why not automate that as well? When it comes to diagnosis or, worse, triage, in my experience you want independent free agents looking at the system all at once. You don't want a warroom-like atmosphere with a single screen but rather n+1 hackers focusing on what their first intuition tells them is the root cause. In a second step you want these hackers to convene and discuss their root cause hypotheses. If necessary, you want them to run experiemnts to confirm these hypotheses. And then you decide the appropriate reaction.
- athenot 5y ago
  Yes, it's a map-reduce algorithm. Muliple people check multiple areas of the system in parallel and then both evidence & rule-outs start to emerge.
- dvtrn 5y ago
  They seem to take inspiration from ER teams or the military It’s probably nothing but overestimation but I feel like I’m seeing more of this later in my career than I did early on, or maybe I’m paying more attention? Whatever it is: past experience (which includes coming from a military family in the states) has taught me to avoid companies that crib unnecessary amounts of jargon, lingo and colloquialisms from the military. Curious if others have noticed or even feel the same and what your experiences have been for feeling similarly?
  - pinko 5y ago
    Agree completely. It's a strong signal that someone has a military cosplay fetish (which very few people with experience in the actual military do), which in turn tends to come along with other dysfunctional traits. It's a warning for me that the person is not likely to be a good vendor, customer, or collaborator.
    - dvtrn 5y ago
      My favorite one was when a superior was explaining a plan to right-size some new machines as we slowly migrated customers onto the appliance, and some particularly aggravating issues we were having with memory consumption that upon inspection and a lot of time spent-made no real sense to us why it was occurring the way it was. "dvtrn you are to take the flank and breach this issue with Paul" And this ran all the way up to the top of the org. Senior leaders were constantly quoting that Jocko Wilink fella. It was...something. My old man (a former Dill Instructor, made for an interesting childhood) found it utterly hilarious when I'd call him up randomly with the latest phrase of the day, uttered by some director or another. To my knowledge, and I sure-damn asked, the only affinity anyone on the executive team had with the military was two of them having buddies who served.
    - pjc50 5y ago
      Yup. It's misplaced machismo, with all that implies.
  - igetspam 5y ago
    I don't know how old you are but my career now exceeds two decades. I definitely see this more now but that's because I institute it. Earlier in my career, we failed at incident management and at ownership. We now share the burden of on-call not just with the operators (sysadmins or old) but also with the people who wrote the code. We've spent a lot of time building better models based on proven methods, quite a few come from work done in high intensity roles paid by tax dollars: risk analysis, disaster recovery, firefighting, command and control, incident management, war games, red teams.
    - 5y ago
- joshuamorton 5y ago
  I agree. I think this particular framing gets things slightly wrong. You want parallelism, but you still need central organization (so that you can have clear delegation) and delegation of work to various researchers. For a complex incident, I've seen 5+ subteams researching various threads of the incident. But, importantly, before any of those subteams take any action, they report to the IC so that two groups don't accidentally take actions that might be good in isolation but are harmful when combined.
- 1123581321 5y ago
  My experience is there’s little conflict between a central conference call or room, and multiple independent investigators, since those investigators need to present and compare their findings somewhere. It would indeed be a mistake to demand everyone look at one high-level view, though. Based on the organization depicted in the article, this would be the “researcher” role, split among multiple people.
jvreagan 5y ago
> nstead I'd be much more impressed to have a "see ya in the afternoon, get some sleep" culture I've led teams for over a decade that have oncall duties. One principle we have lived by is that if you are paged outside of hours, you take off the time you need to not hold a grudge against the team/company. Some people don't need it, some people take a day off, some people sleep in, some people cash in on their next vacation. To each their own according to their needs. Seems to work well. We also swap out oncall in real time if, say, someone gets paged a couple nights in a row.
- deleted 5y ago
  [deleted]
- athenot 5y ago
  Yup, this is important or people will burn out real quick. And when there are major incidents, as IC it's especially important to dismiss people from bridges as early as possible when I know I'm going to need them sooner rather than later the following day. Or swap with a more junior person so that the senior one is nice and fresh for when the next wave is anticipated.
lemax 5y ago
These incident role names are fairly common in product companies these days. I guess you are correct that they do suggest a certain culture around incidents, but in my experience is definitely a good thing. It's a "don't blame people, let's focus on the root cause, get things back up, and figure out how to prevent this next time" sort of thing. People try to meet SLAs and they treat each other like humans. We focus on improving process/frameworks over blaming individual people. And yup, think this comes along with, "incident yesterday was intense, I'm gonna catch up on sleep".
- pm90 5y ago
  I agree with your comment. These names are just ways for teams to delegate different responsibilities w.r.t incident management quickly and in a way that's understood by everyone. Having concrete names for such roles is both a good thing (everyone knows who can make the call for hard decisions) and helps you talk intelligently about the evolution of such roles. e.g. "our Incident Commanders used to spend 15% of their time in p0 incidents, but that has reduced to 10% due to improvements in rollout procedures/runbooks/etc."
zippergz 5y ago
The concepts and terms of incident command are not from the military (or ER as another poster suggested). It's from the fire service and emergency management in general. I don't know if that changes peoples' perceptions and I agree that no amount of terminology changes how exhausting being on call is. But if people are reacting negatively to "military" connotations, I think that is unwarranted. https://en.wikipedia.org/wiki/Incident_Command_System https://en.wikipedia.org/wiki/Incident_Command_System
- nucleardog 5y ago
  I think actually learning what the ICS is _for_ might help people understand a bit better why it's not necessarily just "unnecessary tacticool". It's not just a bunch of important-sounding names for things. ICS, at its core, is a system for helping people self-organize into an effective organization in the face of quickly changing circumstances and emergent problems. Some simple rules are things like: * The most senior/qualified person on-site is generally in charge. (How you determine that kinda varies depending on organization.) * Positions are only created when required. You don't assign people roles unless there's a need for that role. * Positions are split and responsibilities delegated as the span of control increases beyond a set point. * Control should stay as local to the problem as it realistically can while still solving the problem. From there, it goes on to standardized a template hierarchy and defines things like specific colours associated with specific roles so as roles change and chaos ensues, people can continue to operate effectively and in an organized manner. In-person, this means things like the commander/executive roles running around in red vests with their role on the back. If the role changes hands, so does the vest. Some of the roles in that template organization are things like: * The "Public Information Officer" who is responsible for preparing and communicating to the public. This makes a single person responsible to ensure conflicting or confusing messaging is not making its way out. * A "Liason Officer" who is responsible for coordinating with other organizations. This provides another central point of coordination for requests flowing outside of your response. I think we could all imagine how this starts to become valuable in, say, a building collapse scenario with police, fire, EMS, the gas company, search and rescue, emergency social services, etc all on scene. In an IT context, what this means it that, generally, the most senior person online is going to be in charge of receiving reports from people and directing them. If there aren't many people around, they'd generally be pitching in to help as well. As more people show up and the communication and coordination overhead increases, they step out of doing any specific technical work. If enough show up, they may then delegate people out as leaders of specific teams tasked with specific goals (they may also just tell them they're not needed and send them to wait on standby). All roles, including the "Public Information" and "Liason" roles fall to the Incident Commander unless delegated out. At some point, if the requests for reporting from management start interfering with their role as Incident Commander, they delegate that role out. If it turns out the incident is going to require heavy communication or coordination with a vendor, they may delegate out the Liason role to someone else. ICS is probably largely unnecessary if your response never spans larger than the number of people that can effectively communicate in a google meet call, but as you get more and more people involved it contains a lot of valuable lessons and things learned through real world experience in situations much more stressful and dangerous than we ever face that help you effectively manage and coordinate the human resources in response to an incident. (Disclaimer: That's all basically from memory. The city sent me on a ICS, ICS in an emergency operations centre context, and a few more courses a few years back as part of volunteering with an emergency communications group. It's probably 90% accurate.)
dmuth 5y ago
I don't disagree with your post, but one thing I want to mention is the origin of the term "Incident Commander"--it doesn't exist to be cool, but rather derives from how FEMA handles disasters. I suspect its usage in IT became a thing because it was already used in real-life, and it made more sense than creating a new term. If you have two hours, you can take the training that describes the nomenclature behind the Incident Command System, and why it became a thing: https://training.fema.gov/is/courseoverview.aspx?code=is-100.c https://training.fema.gov/is/courseoverview.aspx?code=is-100... This online training takes about 2 hours and is open to the general public. I took it on a Saturday afternoon some years ago and it gave me useful context to why certain things are standardized.
- deleted 5y ago
  [deleted]
- kag0 5y ago
  I was about to post that exact link, as I just completed the course as a prereq for SAR. Here's all the content on one page, I found it less intimidating than the 50 slide format https://emilms.fema.gov/is_0100c/groups/133.html https://emilms.fema.gov/is_0100c/groups/133.html
Sebguer 5y ago
Yeah, the tone of this article is really odd, and like, the bulk of the content is just a narrativization of the incident roles in the Google SRE book. The only 'trick' is running game days?
tetha 5y ago
> We also had a "see ya in the morning" culture. Instead I'd be much more impressed to have a "see ya in the afternoon, get some sleep" culture. German labor laws forbid employees from working 10-13 hours after a long on-call situation after a normal work day, just like that. Add in time compensation, and a bad on-call situation at night easily ends up as the next day off paid. I've found this to take a lot of edge of on-call. Sure, it /sucks/ to get called at 1am and do stuff until 3, but that's a day to sleep in and recover. Maybe hop on a call if the team needs input, but that's optional.
totetsu 5y ago
The worst part is hearing fron your manager the next day that the NOC operator complained about your rude tone of voice when waking up and answering the phone at 3am.
raffraffraff 5y ago
EU working time act means "I'll see you in the afternoon", whether they like it or not.
dreamcompiler 5y ago
Firefighter/EMT here. The fire service has trained on the Incident Command System for decades because it works, not because it's "cool." I didn't know ICS was used in emergency reliability response but it makes perfect sense. Heck, ICS works great for organizing a family camping trip. Like Agile, ICS is a set of principles that work well if you are properly trained in the system. Unlike Agile, ICS has not yet been buzzword-harvested by clueless managers to get promoted, so the concept itself still has some value.

choeger 5y ago

It seems to be a bit of cargo cult, to be honest. They seem to take inspiration from ER teams or the military. I think that this kind of drill helps a lot for cases where you can take a pre-planned route, like deploying that backup server or rerouting traffic. But the obvious question then is: Why not automate that as well? When it comes to diagnosis or, worse, triage, in my experience you want independent free agents looking at the system all at once. You don't want a warroom-like atmosphere with a single screen but rather n+1 hackers focusing on what their first intuition tells them is the root cause. In a second step you want these hackers to convene and discuss their root cause hypotheses. If necessary, you want them to run experiemnts to confirm these hypotheses. And then you decide the appropriate reaction.

athenot 5y ago
Yes, it's a map-reduce algorithm. Muliple people check multiple areas of the system in parallel and then both evidence & rule-outs start to emerge.
dvtrn 5y ago
They seem to take inspiration from ER teams or the military It’s probably nothing but overestimation but I feel like I’m seeing more of this later in my career than I did early on, or maybe I’m paying more attention? Whatever it is: past experience (which includes coming from a military family in the states) has taught me to avoid companies that crib unnecessary amounts of jargon, lingo and colloquialisms from the military. Curious if others have noticed or even feel the same and what your experiences have been for feeling similarly?
- pinko 5y ago
  Agree completely. It's a strong signal that someone has a military cosplay fetish (which very few people with experience in the actual military do), which in turn tends to come along with other dysfunctional traits. It's a warning for me that the person is not likely to be a good vendor, customer, or collaborator.
  - dvtrn 5y ago
    My favorite one was when a superior was explaining a plan to right-size some new machines as we slowly migrated customers onto the appliance, and some particularly aggravating issues we were having with memory consumption that upon inspection and a lot of time spent-made no real sense to us why it was occurring the way it was. "dvtrn you are to take the flank and breach this issue with Paul" And this ran all the way up to the top of the org. Senior leaders were constantly quoting that Jocko Wilink fella. It was...something. My old man (a former Dill Instructor, made for an interesting childhood) found it utterly hilarious when I'd call him up randomly with the latest phrase of the day, uttered by some director or another. To my knowledge, and I sure-damn asked, the only affinity anyone on the executive team had with the military was two of them having buddies who served.
  - pjc50 5y ago
    Yup. It's misplaced machismo, with all that implies.
- igetspam 5y ago
  I don't know how old you are but my career now exceeds two decades. I definitely see this more now but that's because I institute it. Earlier in my career, we failed at incident management and at ownership. We now share the burden of on-call not just with the operators (sysadmins or old) but also with the people who wrote the code. We've spent a lot of time building better models based on proven methods, quite a few come from work done in high intensity roles paid by tax dollars: risk analysis, disaster recovery, firefighting, command and control, incident management, war games, red teams.
  - 5y ago
joshuamorton 5y ago
I agree. I think this particular framing gets things slightly wrong. You want parallelism, but you still need central organization (so that you can have clear delegation) and delegation of work to various researchers. For a complex incident, I've seen 5+ subteams researching various threads of the incident. But, importantly, before any of those subteams take any action, they report to the IC so that two groups don't accidentally take actions that might be good in isolation but are harmful when combined.
1123581321 5y ago
My experience is there’s little conflict between a central conference call or room, and multiple independent investigators, since those investigators need to present and compare their findings somewhere. It would indeed be a mistake to demand everyone look at one high-level view, though. Based on the organization depicted in the article, this would be the “researcher” role, split among multiple people.

jvreagan 5y ago

> nstead I'd be much more impressed to have a "see ya in the afternoon, get some sleep" culture I've led teams for over a decade that have oncall duties. One principle we have lived by is that if you are paged outside of hours, you take off the time you need to not hold a grudge against the team/company. Some people don't need it, some people take a day off, some people sleep in, some people cash in on their next vacation. To each their own according to their needs. Seems to work well. We also swap out oncall in real time if, say, someone gets paged a couple nights in a row.

deleted 5y ago
[deleted]
athenot 5y ago
Yup, this is important or people will burn out real quick. And when there are major incidents, as IC it's especially important to dismiss people from bridges as early as possible when I know I'm going to need them sooner rather than later the following day. Or swap with a more junior person so that the senior one is nice and fresh for when the next wave is anticipated.

lemax 5y ago

These incident role names are fairly common in product companies these days. I guess you are correct that they do suggest a certain culture around incidents, but in my experience is definitely a good thing. It's a "don't blame people, let's focus on the root cause, get things back up, and figure out how to prevent this next time" sort of thing. People try to meet SLAs and they treat each other like humans. We focus on improving process/frameworks over blaming individual people. And yup, think this comes along with, "incident yesterday was intense, I'm gonna catch up on sleep".

pm90 5y ago
I agree with your comment. These names are just ways for teams to delegate different responsibilities w.r.t incident management quickly and in a way that's understood by everyone. Having concrete names for such roles is both a good thing (everyone knows who can make the call for hard decisions) and helps you talk intelligently about the evolution of such roles. e.g. "our Incident Commanders used to spend 15% of their time in p0 incidents, but that has reduced to 10% due to improvements in rollout procedures/runbooks/etc."

zippergz 5y ago

The concepts and terms of incident command are not from the military (or ER as another poster suggested). It's from the fire service and emergency management in general. I don't know if that changes peoples' perceptions and I agree that no amount of terminology changes how exhausting being on call is. But if people are reacting negatively to "military" connotations, I think that is unwarranted. https://en.wikipedia.org/wiki/Incident_Command_System https://en.wikipedia.org/wiki/Incident_Command_System

nucleardog 5y ago
I think actually learning what the ICS is _for_ might help people understand a bit better why it's not necessarily just "unnecessary tacticool". It's not just a bunch of important-sounding names for things. ICS, at its core, is a system for helping people self-organize into an effective organization in the face of quickly changing circumstances and emergent problems. Some simple rules are things like: * The most senior/qualified person on-site is generally in charge. (How you determine that kinda varies depending on organization.) * Positions are only created when required. You don't assign people roles unless there's a need for that role. * Positions are split and responsibilities delegated as the span of control increases beyond a set point. * Control should stay as local to the problem as it realistically can while still solving the problem. From there, it goes on to standardized a template hierarchy and defines things like specific colours associated with specific roles so as roles change and chaos ensues, people can continue to operate effectively and in an organized manner. In-person, this means things like the commander/executive roles running around in red vests with their role on the back. If the role changes hands, so does the vest. Some of the roles in that template organization are things like: * The "Public Information Officer" who is responsible for preparing and communicating to the public. This makes a single person responsible to ensure conflicting or confusing messaging is not making its way out. * A "Liason Officer" who is responsible for coordinating with other organizations. This provides another central point of coordination for requests flowing outside of your response. I think we could all imagine how this starts to become valuable in, say, a building collapse scenario with police, fire, EMS, the gas company, search and rescue, emergency social services, etc all on scene. In an IT context, what this means it that, generally, the most senior person online is going to be in charge of receiving reports from people and directing them. If there aren't many people around, they'd generally be pitching in to help as well. As more people show up and the communication and coordination overhead increases, they step out of doing any specific technical work. If enough show up, they may then delegate people out as leaders of specific teams tasked with specific goals (they may also just tell them they're not needed and send them to wait on standby). All roles, including the "Public Information" and "Liason" roles fall to the Incident Commander unless delegated out. At some point, if the requests for reporting from management start interfering with their role as Incident Commander, they delegate that role out. If it turns out the incident is going to require heavy communication or coordination with a vendor, they may delegate out the Liason role to someone else. ICS is probably largely unnecessary if your response never spans larger than the number of people that can effectively communicate in a google meet call, but as you get more and more people involved it contains a lot of valuable lessons and things learned through real world experience in situations much more stressful and dangerous than we ever face that help you effectively manage and coordinate the human resources in response to an incident. (Disclaimer: That's all basically from memory. The city sent me on a ICS, ICS in an emergency operations centre context, and a few more courses a few years back as part of volunteering with an emergency communications group. It's probably 90% accurate.)

dmuth 5y ago

I don't disagree with your post, but one thing I want to mention is the origin of the term "Incident Commander"--it doesn't exist to be cool, but rather derives from how FEMA handles disasters. I suspect its usage in IT became a thing because it was already used in real-life, and it made more sense than creating a new term. If you have two hours, you can take the training that describes the nomenclature behind the Incident Command System, and why it became a thing: https://training.fema.gov/is/courseoverview.aspx?code=is-100.c https://training.fema.gov/is/courseoverview.aspx?code=is-100... This online training takes about 2 hours and is open to the general public. I took it on a Saturday afternoon some years ago and it gave me useful context to why certain things are standardized.

deleted 5y ago
[deleted]
kag0 5y ago
I was about to post that exact link, as I just completed the course as a prereq for SAR. Here's all the content on one page, I found it less intimidating than the 50 slide format https://emilms.fema.gov/is_0100c/groups/133.html https://emilms.fema.gov/is_0100c/groups/133.html

Sebguer 5y ago

Yeah, the tone of this article is really odd, and like, the bulk of the content is just a narrativization of the incident roles in the Google SRE book. The only 'trick' is running game days?

tetha 5y ago

> We also had a "see ya in the morning" culture. Instead I'd be much more impressed to have a "see ya in the afternoon, get some sleep" culture. German labor laws forbid employees from working 10-13 hours after a long on-call situation after a normal work day, just like that. Add in time compensation, and a bad on-call situation at night easily ends up as the next day off paid. I've found this to take a lot of edge of on-call. Sure, it /sucks/ to get called at 1am and do stuff until 3, but that's a day to sleep in and recover. Maybe hop on a call if the team needs input, but that's optional.

totetsu 5y ago

The worst part is hearing fron your manager the next day that the NOC operator complained about your rude tone of voice when waking up and answering the phone at 3am.

raffraffraff 5y ago

EU working time act means "I'll see you in the afternoon", whether they like it or not.

dreamcompiler 5y ago

Firefighter/EMT here. The fire service has trained on the Incident Command System for decades because it works, not because it's "cool." I didn't know ICS was used in emergency reliability response but it makes perfect sense. Heck, ICS works great for organizing a family camping trip. Like Agile, ICS is a set of principles that work well if you are properly trained in the system. Unlike Agile, ICS has not yet been buzzword-harvested by clueless managers to get promoted, so the concept itself still has some value.