4 ms·

Haven't been involved in realtime shit-hits-the-fan ops for some time now (phew). Are devops/SRE/etc people using voice comms these days during high impact inc

by zgluck 3y ago

Haven't been involved in realtime shit-hits-the-fan ops for some time now (phew). Are devops/SRE/etc people using voice comms these days during high impact incidents or is text (irc/slack/etc) still preferred?

foobarbecue 3y ago
Dunno. I'm talking about working Curiosity and Ingenuity ops (Mars).
- zgluck 3y ago
  :-) Cool. I guess I assumed you were deploying overly complicated glorified CRUD apps like (most of) the rest of us. So what do you use for comms during high impact incidents? Are you all in the same room?
  - foobarbecue 3y ago
    https://dataverse.jpl.nasa.gov/file.xhtml?fileId=74511&version=1.0 https://dataverse.jpl.nasa.gov/file.xhtml?fileId=74511&versi... is still pretty accurate. Uplink is still mostly work from home for Curiosity and Ingenuity. And as it happens... I do CRUD web apps too, offically about 20% of my time. Including that one. And for those we use text.
    - zgluck 3y ago
      It would be really interesting to learn how you value text vs voice comms, for instance.
      - foobarbecue 3y ago
        We rely on both, and I think we've got the mix about right. Text for relatively simple things, when numbers and other identifiers need to be transmitted accurately, voice / video for complex discussions that require screen sharing and / or when there are multiple opinions that might need to be reconciled.
  - foobarbecue 3y ago
    Better link: https://doi.org/10.1109/AERO50100.2021.9438370 https://doi.org/10.1109/AERO50100.2021.9438370
- dylan604 3y ago
  I'm surprised that's not already an thing in that ops center. Maybe being an outsider I've built up an unrealistic aura of awe around NASA/JPL type environments where those places would be quiet as a church unless something like landing or some other epic accomplishment where everyone was cheering. Otherwise, I'd expect everyone to know that when the shit hits the fan, everyone is hunkered down doing what needs to be done. Does the long lag between issuing commands kind of break that?
  - foobarbecue 3y ago
    There's a well-choreographed dance of meetings (with polls, e.g. go / no go) and head-down sequence writing. Meetings have a pretty rigid structure and there are zero off-topic comments. But the time for getting the actual sequencing done often has chatter with varying amounts of on-topic-ness. Generally, people are really great at not distracting each other-- I've only occasionally had an issue and when I bring it up, chatter ends. It's just that having a term for it would make it easier to bring up. "Socialize with your colleagues" is actually an item on one of our checklists!
    - deleted 3y ago
      [deleted]
    - dylan604 3y ago
      Do you feel that something as long running as Curiosity that things become a little more lax as everyone gets to a level of complacency? I could see where that might happen, but at the same time the concept of I'm controlling something on the surface of another planet is hard to forget. Basically, super curious about the day to day of a job I'm very jealous of in a very respectful way
      - foobarbecue 3y ago
        Interesting question about complacency. Hard for me to speak objectively on this since I'm too close to it to be unbiased, but I'll try. I actually think the level of acceptable risk is gradually dropping. Each time something goes wrong, we add procedures. It's much easier to add rules than to remove them. I do think the risk tolerance of individuals on the project does tend to gradually increase with experience, but we have fairly high turnover. It's common for MSL roles to have a year or more of training before certification during which you spend a lot of time focusing on what can go wrong, so new people definitely trend cautious.
inopinatus 3y ago
That’s a yes. I have personally been on public cloud major incident calls. They are strictly on-topic only and generally only the incident coordinator, their offsider, and one or two SMEs, are authorised to speak. There may be hundreds or even thousands of listeners around the globe, most of which are there to fan-out status to impacted internal teams and major customers. However there will also be multiple text-based side-channels going on with the involved service teams.
- liotier 3y ago
  Telco guy here, similar story. Our standard procedure dictates who is on the crisis call, with increasingly high ranks being sucked in as the incident escalates the severity scale by meeting objective criteria - for the highest levels of outage (think "no more phone service in the whole country" or "ministry of defence lost all its fiber links"), it can theoretically mean waking up the CEO in the middle of the night. Representatives of involved teams may come in and out of the call but usually stay on to maintain liaison.

by zgluck 3y ago

foobarbecue 3y ago

Dunno. I'm talking about working Curiosity and Ingenuity ops (Mars).

zgluck 3y ago
:-) Cool. I guess I assumed you were deploying overly complicated glorified CRUD apps like (most of) the rest of us. So what do you use for comms during high impact incidents? Are you all in the same room?
- foobarbecue 3y ago
  https://dataverse.jpl.nasa.gov/file.xhtml?fileId=74511&version=1.0 https://dataverse.jpl.nasa.gov/file.xhtml?fileId=74511&versi... is still pretty accurate. Uplink is still mostly work from home for Curiosity and Ingenuity. And as it happens... I do CRUD web apps too, offically about 20% of my time. Including that one. And for those we use text.
  - zgluck 3y ago
    It would be really interesting to learn how you value text vs voice comms, for instance.
    - foobarbecue 3y ago
      We rely on both, and I think we've got the mix about right. Text for relatively simple things, when numbers and other identifiers need to be transmitted accurately, voice / video for complex discussions that require screen sharing and / or when there are multiple opinions that might need to be reconciled.
- foobarbecue 3y ago
  Better link: https://doi.org/10.1109/AERO50100.2021.9438370 https://doi.org/10.1109/AERO50100.2021.9438370
dylan604 3y ago
I'm surprised that's not already an thing in that ops center. Maybe being an outsider I've built up an unrealistic aura of awe around NASA/JPL type environments where those places would be quiet as a church unless something like landing or some other epic accomplishment where everyone was cheering. Otherwise, I'd expect everyone to know that when the shit hits the fan, everyone is hunkered down doing what needs to be done. Does the long lag between issuing commands kind of break that?
- foobarbecue 3y ago
  There's a well-choreographed dance of meetings (with polls, e.g. go / no go) and head-down sequence writing. Meetings have a pretty rigid structure and there are zero off-topic comments. But the time for getting the actual sequencing done often has chatter with varying amounts of on-topic-ness. Generally, people are really great at not distracting each other-- I've only occasionally had an issue and when I bring it up, chatter ends. It's just that having a term for it would make it easier to bring up. "Socialize with your colleagues" is actually an item on one of our checklists!
  - deleted 3y ago
    [deleted]
  - dylan604 3y ago
    Do you feel that something as long running as Curiosity that things become a little more lax as everyone gets to a level of complacency? I could see where that might happen, but at the same time the concept of I'm controlling something on the surface of another planet is hard to forget. Basically, super curious about the day to day of a job I'm very jealous of in a very respectful way
    - foobarbecue 3y ago
      Interesting question about complacency. Hard for me to speak objectively on this since I'm too close to it to be unbiased, but I'll try. I actually think the level of acceptable risk is gradually dropping. Each time something goes wrong, we add procedures. It's much easier to add rules than to remove them. I do think the risk tolerance of individuals on the project does tend to gradually increase with experience, but we have fairly high turnover. It's common for MSL roles to have a year or more of training before certification during which you spend a lot of time focusing on what can go wrong, so new people definitely trend cautious.

inopinatus 3y ago

That’s a yes. I have personally been on public cloud major incident calls. They are strictly on-topic only and generally only the incident coordinator, their offsider, and one or two SMEs, are authorised to speak. There may be hundreds or even thousands of listeners around the globe, most of which are there to fan-out status to impacted internal teams and major customers. However there will also be multiple text-based side-channels going on with the involved service teams.

liotier 3y ago
Telco guy here, similar story. Our standard procedure dictates who is on the crisis call, with increasingly high ranks being sucked in as the incident escalates the severity scale by meeting objective criteria - for the highest levels of outage (think "no more phone service in the whole country" or "ministry of defence lost all its fiber links"), it can theoretically mean waking up the CEO in the middle of the night. Representatives of involved teams may come in and out of the call but usually stay on to maintain liaison.