4 ms·
I've worked in >1 companies where the SLA requires hourly updates for incidents, and they have to have substance beyond "still fixing." This is in large scale
by Jenk 4y ago
I've worked in >1 companies where the SLA requires hourly updates for incidents, and they have to have substance beyond "still fixing."
This is in large scale finance and/or investment sectors where it is immeasurably important to keep clients (both "those paying us" but also the "those integrating with us" types) well informed of progress so as not to cause market panic.
Sometimes communicating is worth it for interrupting the team.
- emeraldd 4y agoThere's communication by interruption and there's observation. If you have a situation where there need to be hourly updates, there should be a person explicitly tasked with observing and noting what's going on to be able to provide those updates with minimal impact to the group that's actually working the problem. Unfortunately, that means the observer has to be able to understand and follow what the engineers are doing while staying out of the way as much as possible. Tricky, but something you really need if for no other reason than to communicate and look for any holes that the team/person working the problem may not have thought of in the interim.
- roland35 4y agoYes, this role is defined as the "journalist" at my company and is an engineer tasked with writing status updates and the post-mortem report. It is important to have this role assigned from the start so that expectations are clear!
- bluGill 4y agoIf that is the case you should pair program, where one of the pair's only job is to sit behind the engineer doing the job and listen in, then take the meetings. This needs to be a full engineer who is fully capable of doing the work, but instead is just understanding the work, but not doing any. If the main engineer has a 'family emergency' your communication engineer has enough context to take over, while another engineer is assigned to communication.
- antod 4y agoAgreed. That was how I've seen it run well. eg we'd create 2 channels for the incident - the first was for the technical team trying to fix it and would have quite a bit of noise as debugging went on. The second was for more senior managers and the client/account people to coordinate comms with customers. The only manager in the first group was the teams direct technical manager or their eg Tech Lead type of role. Their job was to not work on the fix but to communicate what they were observing to the 2nd group who would deal with the customers. Smaller companies though are a bit less structured, and have to wing it a bit more depending on circumstances.
- jeffparsons 4y agoAssigning a "communications officer" at the start of an incident seems to work well. They are a person with the same technical skills as the people actively working on the problem, but their job is primarily to observe and filter/transform communications to and from the team. A nice side benefit is that having someone who isn't actually responsible for fixing things seems to free up their mind to notice things that their hands-on-keyboards colleagues missed.
- dilyevsky 4y agoClearly you need non-technical manager to communicate status to the customers. You know, because they have people skills
- pmarreck 4y agoI knew this conversation was going to eventually end up in an Office Space reference...
- tremon 4y agoSometimes communicating is worth it for interrupting the team Perhaps, but the first thing any emergency responder will tell you is that there should be one incident commander, and that all communication and allocation of resources should go through that one person. And that the person with the incident commander role should not carry out any operational duties besides coordinate and control, because those other tasks require the same mental faculties, but use them in a different way, making them less effective in either. If there is a requirement to update customers affected by an incident, the company should have processes in place to streamline the flow of information without interrupting the team. And the team responsible for updating customers should only communicate with the incident commander, and absolutely steer clear of the operational responders. If specific details are required, it falls to the incident commander to gather those details from the team, not to outside people running interference.
- Consultant32452 4y agoThe key component in this system that understands human psychology is to set reasonable expectations. Most places don't have that kind of SLA, but when an event occurs you can set an expectation of when the next communication will be and as long as it's not crazy, people will go along with it. But if you just leave people hanging, not knowing when/if an update is coming, they have nothing to do but panic about the lack of new information.
- PragmaticPulp 4y ago> Sometimes communicating is worth it for interrupting the team. If you need constant updates from someone who knows the subject material, you need to nominate a single person who knows the material to take point on communication. They can help in between communication updates, but they can’t be tasked with leading or owning the work. The worst way to do it is to have a manager who can’t understand the subject material themselves, then let them interrupt the entire team over and over again. Then the team has to stop and educate the manager about what needs to be communicated so they can relay it to someone else. Having non-technical managers relaying information back and forth from the entire team is always a mistake.
- ilyt 4y agoScheduled communication is far easier to handle than random interruptions. If your target is to get update every hour you just dedicate a person during incident to communicate with the "outside" and make rest of the team post the updates on shared chat, which you want anyway if you don't want people to get on eachother's toes