4 ms·

Excellent article. Taught me about the "sterile cockpit rule," which I think might be really useful at my work. Sometimes people start chatting at critical mome

Excellent article. Taught me about the "sterile cockpit rule," which I think might be really useful at my work. Sometimes people start chatting at critical moments during operations and I think I might introduce the concept of "sterile cockpit" during those moments to get people's focus back.

zgluck 3y ago
Haven't been involved in realtime shit-hits-the-fan ops for some time now (phew). Are devops/SRE/etc people using voice comms these days during high impact incidents or is text (irc/slack/etc) still preferred?
- foobarbecue 3y ago
  Dunno. I'm talking about working Curiosity and Ingenuity ops (Mars).
  - zgluck 3y ago
    :-) Cool. I guess I assumed you were deploying overly complicated glorified CRUD apps like (most of) the rest of us. So what do you use for comms during high impact incidents? Are you all in the same room?
    - foobarbecue 3y ago
      https://dataverse.jpl.nasa.gov/file.xhtml?fileId=74511&version=1.0 https://dataverse.jpl.nasa.gov/file.xhtml?fileId=74511&versi... is still pretty accurate. Uplink is still mostly work from home for Curiosity and Ingenuity. And as it happens... I do CRUD web apps too, offically about 20% of my time. Including that one. And for those we use text.
      - zgluck 3y ago
        It would be really interesting to learn how you value text vs voice comms, for instance.
        foobarbecue 3y ago
        We rely on both, and I think we've got the mix about right. Text for relatively simple things, when numbers and other identifiers need to be transmitted accurately, voice / video for complex discussions that require screen sharing and / or when there are multiple opinions that might need to be reconciled.
    - foobarbecue 3y ago
      Better link: https://doi.org/10.1109/AERO50100.2021.9438370 https://doi.org/10.1109/AERO50100.2021.9438370
  - dylan604 3y ago
    I'm surprised that's not already an thing in that ops center. Maybe being an outsider I've built up an unrealistic aura of awe around NASA/JPL type environments where those places would be quiet as a church unless something like landing or some other epic accomplishment where everyone was cheering. Otherwise, I'd expect everyone to know that when the shit hits the fan, everyone is hunkered down doing what needs to be done. Does the long lag between issuing commands kind of break that?
    - foobarbecue 3y ago
      There's a well-choreographed dance of meetings (with polls, e.g. go / no go) and head-down sequence writing. Meetings have a pretty rigid structure and there are zero off-topic comments. But the time for getting the actual sequencing done often has chatter with varying amounts of on-topic-ness. Generally, people are really great at not distracting each other-- I've only occasionally had an issue and when I bring it up, chatter ends. It's just that having a term for it would make it easier to bring up. "Socialize with your colleagues" is actually an item on one of our checklists!
      - deleted 3y ago
        [deleted]
      - dylan604 3y ago
        Do you feel that something as long running as Curiosity that things become a little more lax as everyone gets to a level of complacency? I could see where that might happen, but at the same time the concept of I'm controlling something on the surface of another planet is hard to forget. Basically, super curious about the day to day of a job I'm very jealous of in a very respectful way
        foobarbecue 3y ago
        Interesting question about complacency. Hard for me to speak objectively on this since I'm too close to it to be unbiased, but I'll try. I actually think the level of acceptable risk is gradually dropping. Each time something goes wrong, we add procedures. It's much easier to add rules than to remove them. I do think the risk tolerance of individuals on the project does tend to gradually increase with experience, but we have fairly high turnover. It's common for MSL roles to have a year or more of training before certification during which you spend a lot of time focusing on what can go wrong, so new people definitely trend cautious.
- inopinatus 3y ago
  That’s a yes. I have personally been on public cloud major incident calls. They are strictly on-topic only and generally only the incident coordinator, their offsider, and one or two SMEs, are authorised to speak. There may be hundreds or even thousands of listeners around the globe, most of which are there to fan-out status to impacted internal teams and major customers. However there will also be multiple text-based side-channels going on with the involved service teams.
  - liotier 3y ago
    Telco guy here, similar story. Our standard procedure dictates who is on the crisis call, with increasingly high ranks being sucked in as the incident escalates the severity scale by meeting objective criteria - for the highest levels of outage (think "no more phone service in the whole country" or "ministry of defence lost all its fiber links"), it can theoretically mean waking up the CEO in the middle of the night. Representatives of involved teams may come in and out of the call but usually stay on to maintain liaison.
duxup 3y ago
When I used to work networking tech support we had rules about how if there was a P1 network down situation that sales and account managers got their own conference call… and engineers had their own. (obviously we would sync up at some point, eventually) It was also understood that non technical managers and etc could not wander over to the tech support floor. It was absolutely critical in order to maintain good technical communication and etc. No drama, just matter of fact check lists and technical discussions. A big bank could have all their ATMs down (sorry I did that once) and the discussion was all technical and cool and calm. Our technical staff customers loved it too as their management were on the other call too ;)
Terr_ 3y ago
I imagine such rules are easier to apply when it is a singular context and physical delineation of space. It takes advantage of our subconscious human rules of behavior. An office-example might be if you had the super-serious conference room that is exclusively used for super-serious things. The physical context would affect how likely people are to follow the desired norms.
- Kye 3y ago
  “Gentlemen, you can’t fight in here! This is the War Room!”

zgluck 3y ago

Haven't been involved in realtime shit-hits-the-fan ops for some time now (phew). Are devops/SRE/etc people using voice comms these days during high impact incidents or is text (irc/slack/etc) still preferred?

foobarbecue 3y ago
Dunno. I'm talking about working Curiosity and Ingenuity ops (Mars).
- zgluck 3y ago
  :-) Cool. I guess I assumed you were deploying overly complicated glorified CRUD apps like (most of) the rest of us. So what do you use for comms during high impact incidents? Are you all in the same room?
  - foobarbecue 3y ago
    https://dataverse.jpl.nasa.gov/file.xhtml?fileId=74511&version=1.0 https://dataverse.jpl.nasa.gov/file.xhtml?fileId=74511&versi... is still pretty accurate. Uplink is still mostly work from home for Curiosity and Ingenuity. And as it happens... I do CRUD web apps too, offically about 20% of my time. Including that one. And for those we use text.
    - zgluck 3y ago
      It would be really interesting to learn how you value text vs voice comms, for instance.
      - foobarbecue 3y ago
        We rely on both, and I think we've got the mix about right. Text for relatively simple things, when numbers and other identifiers need to be transmitted accurately, voice / video for complex discussions that require screen sharing and / or when there are multiple opinions that might need to be reconciled.
  - foobarbecue 3y ago
    Better link: https://doi.org/10.1109/AERO50100.2021.9438370 https://doi.org/10.1109/AERO50100.2021.9438370
- dylan604 3y ago
  I'm surprised that's not already an thing in that ops center. Maybe being an outsider I've built up an unrealistic aura of awe around NASA/JPL type environments where those places would be quiet as a church unless something like landing or some other epic accomplishment where everyone was cheering. Otherwise, I'd expect everyone to know that when the shit hits the fan, everyone is hunkered down doing what needs to be done. Does the long lag between issuing commands kind of break that?
  - foobarbecue 3y ago
    There's a well-choreographed dance of meetings (with polls, e.g. go / no go) and head-down sequence writing. Meetings have a pretty rigid structure and there are zero off-topic comments. But the time for getting the actual sequencing done often has chatter with varying amounts of on-topic-ness. Generally, people are really great at not distracting each other-- I've only occasionally had an issue and when I bring it up, chatter ends. It's just that having a term for it would make it easier to bring up. "Socialize with your colleagues" is actually an item on one of our checklists!
    - deleted 3y ago
      [deleted]
    - dylan604 3y ago
      Do you feel that something as long running as Curiosity that things become a little more lax as everyone gets to a level of complacency? I could see where that might happen, but at the same time the concept of I'm controlling something on the surface of another planet is hard to forget. Basically, super curious about the day to day of a job I'm very jealous of in a very respectful way
      - foobarbecue 3y ago
        Interesting question about complacency. Hard for me to speak objectively on this since I'm too close to it to be unbiased, but I'll try. I actually think the level of acceptable risk is gradually dropping. Each time something goes wrong, we add procedures. It's much easier to add rules than to remove them. I do think the risk tolerance of individuals on the project does tend to gradually increase with experience, but we have fairly high turnover. It's common for MSL roles to have a year or more of training before certification during which you spend a lot of time focusing on what can go wrong, so new people definitely trend cautious.
inopinatus 3y ago
That’s a yes. I have personally been on public cloud major incident calls. They are strictly on-topic only and generally only the incident coordinator, their offsider, and one or two SMEs, are authorised to speak. There may be hundreds or even thousands of listeners around the globe, most of which are there to fan-out status to impacted internal teams and major customers. However there will also be multiple text-based side-channels going on with the involved service teams.
- liotier 3y ago
  Telco guy here, similar story. Our standard procedure dictates who is on the crisis call, with increasingly high ranks being sucked in as the incident escalates the severity scale by meeting objective criteria - for the highest levels of outage (think "no more phone service in the whole country" or "ministry of defence lost all its fiber links"), it can theoretically mean waking up the CEO in the middle of the night. Representatives of involved teams may come in and out of the call but usually stay on to maintain liaison.

duxup 3y ago

When I used to work networking tech support we had rules about how if there was a P1 network down situation that sales and account managers got their own conference call… and engineers had their own. (obviously we would sync up at some point, eventually) It was also understood that non technical managers and etc could not wander over to the tech support floor. It was absolutely critical in order to maintain good technical communication and etc. No drama, just matter of fact check lists and technical discussions. A big bank could have all their ATMs down (sorry I did that once) and the discussion was all technical and cool and calm. Our technical staff customers loved it too as their management were on the other call too ;)

Terr_ 3y ago

I imagine such rules are easier to apply when it is a singular context and physical delineation of space. It takes advantage of our subconscious human rules of behavior. An office-example might be if you had the super-serious conference room that is exclusively used for super-serious things. The physical context would affect how likely people are to follow the desired norms.

Kye 3y ago
“Gentlemen, you can’t fight in here! This is the War Room!”