4 ms·

We are discussing this at the place I work at the moment. There is something I've never known the answer to, perhaps someone here who knows more about incident

We are discussing this at the place I work at the moment. There is something I've never known the answer to, perhaps someone here who knows more about incidents can help. I have joined a small team. There are perhaps 2-3 devs who can actually fix issues. There are lots of issues, e.g. one incident a week. We're obviously trying to fix the software but every time something goes wrong it's a new problem. We have 100+ microservices interacting and 1k req/second, on a 20-year old codebase where most of the services are written by people who are no longer at the company, so although we're trying and making improvements, and I'm doing my best to help, it's not going to happen overnight. Which brings us back to incident support. We're too small a team to have an on-call rota (i.e. with 3 people everyone would have to be on call once every 3 nights, apart from when people are on vacation or sick when the remaining people would have to be on-call more). So the management's idea is just that everyone have slack on their phone and whoever is available jumps in either when an alert goes off, or when someone (e.g. support) writes something isn't working. "Professional jobs aren't just 9-5, they do involve overtime sometimes". To be this just feels like being permanently on-call, which I don't want (burnout risk etc.). It might be different if there was an incident once a year but it's much more frequent than that. So I know what I think we shouldn't do, but what should I advise the company that we should do? We can't fix all the issues overnight, every incident seems different, we have too few people to have a rota, the software is too complex to outside operations to an external team (e.g. last fix last night was for one of my colleagues to deploy a quick hack to the codebase to turn off one feature, then build the code and deploy it), yet it's a international consumer website that should be available 24/7.

sigwinch28 4y ago
1. Collect data. It will help your case when you talk to management. How many incidents, when, what they relate to, how much of team time is spent on firefighting, etc. 2. Read Google’s SRE book for ideas. For example, read about error budgets in this chapter: https://sre.google/sre-book/embracing-risk/ https://sre.google/sre-book/embracing-risk/ If the goal is to have a reliable system, then the ultimate creators of unreliable systems must be made to feel the pain themselves. For example, product can be made to “feel the pain” by not being allowed to have any new features while an engineering team fixes user-facing or business-critical bugs ultimately caused by an endless plethora of new features being requested by product on short timescales. In one word: backpressure. Of course, this relies on product and/or management buying into the idea, which is why data can help you argue your case. The rest of your organisation might be more receptive if you can show them how much time is being spent on incidents and bug whack-a-mole.
- raffraffraff 4y ago
  I agree with collecting data. Monitoring tooling is extremely important. With microservices you need metrics (requests, errors, latency), logs, traces. Every service will have specific signals that the service owner should understand. The SRE book is an OK idea but to be honest, it's hard to get developers to focus on Google-level theory of reliable engineering via error budgets when they're so operationally immature that nobody is oncall (so everybody is oncall). Sounds like the gap is too wide. If these alerts are 50% "customer reported an incident" instead of a SLI, then that SRE book could spawn a cargo cult that creates work for everybody and achieves nothing because the gap is too wide. I've seen clueless managers latch onto theory that's too far ahead of the stone-age. Like "Hey guys, this month I want everybody to record their SLOs", when nobody is even recording metrics because they don't know how, or what to record. I'd do extremely boring stuff like keep a spreadsheet of incidents with columns for severity, trigger (eg release, operator error, customer load, hidden bug, system error), root cause (eg internal software bug, bad config, system resources etc). If patterns emerge, they should point to gaps in knowledge, testing, automation, monitoring etc. If there's no pattern at all, then your engineering org is in a bad way. When an engineering department is extremely dysfunctional, it needs external help. Or sometimes it needs a clueless manager / lead to quit, allowing suppressed talent too take over and fix stuff.
Normal_gaussian 4y ago
On call rota - 1 week on, n weeks off, but make sure its compensated. Primary duty of on call is triage, pushing all non-critical issues to work hours and only bringing issues they deal with through mitigation and not resolution.
evnsio 4y ago
When you say there's 2-3 devs who can actually fix issues, is that because there are only 2-3 devs, or do you have more people available but they don't know how to fix? Assuming the latter, my suggestion would be: 1) Setup a weekly on-call rotation, where one of the 3 people who can fix issues is dedicated on-call (this means the other 2 can properly switch off) 2) Find some other folks who might not be able to support immediately, but are happy to learn. 3) Configure a shadow rotation so there's always a primary responder who know what they're doing and a shadow responder who can learn on the job. 4) Make it clear the shadow person has the same expectations in terms of turning up, but their job is to ask questions, write things down, etc. I did this in a former job (and wrote about it here https://monzo.com/blog/2018/09/20/on-call https://monzo.com/blog/2018/09/20/on-call) and it worked pretty well. After a few months we had a healthy rotation of 8 on-callers, and a waitlist of shadow folks who wanted to join too. This advice obviously came from a point of near-zero context, so if you'd like to chat more I'd be happy to!
- sokoloff 4y ago
  Did something similar and that program turned into a milestone on many engineers’ promotion journey. (It wasn’t mandatory, but most everyone who made a certain level had successfully completed the program because you learned so much in and got such broad exposure from it.)
janci 4y ago
Define a SLA. Classify incidents by priority. Not everything must be solved immediately. Hire a developer from different time zone to cover your night time by his day time. Make your architecture more resilient. Identify most impactful improvements and stop implementation of new features until they are solved. Compensate well for the overtime. Ultimately, run away.
tomerbd 4y ago
The one who handled an error external to working hours on day x gets a day off on day x+1 otherwise you will burn out.
raffraffraff 4y ago
> So the management's idea is just that everyone have slack on their phone and whoever is available jumps in either when an alert goes off Is this a 24x7 thing? If so, this is the worst idea ever. I would quit immediately. Screw that manager. Honestly. This is clueless, and it shows a lack of care for the whole team.

sigwinch28 4y ago

1. Collect data. It will help your case when you talk to management. How many incidents, when, what they relate to, how much of team time is spent on firefighting, etc. 2. Read Google’s SRE book for ideas. For example, read about error budgets in this chapter: https://sre.google/sre-book/embracing-risk/ https://sre.google/sre-book/embracing-risk/ If the goal is to have a reliable system, then the ultimate creators of unreliable systems must be made to feel the pain themselves. For example, product can be made to “feel the pain” by not being allowed to have any new features while an engineering team fixes user-facing or business-critical bugs ultimately caused by an endless plethora of new features being requested by product on short timescales. In one word: backpressure. Of course, this relies on product and/or management buying into the idea, which is why data can help you argue your case. The rest of your organisation might be more receptive if you can show them how much time is being spent on incidents and bug whack-a-mole.

raffraffraff 4y ago
I agree with collecting data. Monitoring tooling is extremely important. With microservices you need metrics (requests, errors, latency), logs, traces. Every service will have specific signals that the service owner should understand. The SRE book is an OK idea but to be honest, it's hard to get developers to focus on Google-level theory of reliable engineering via error budgets when they're so operationally immature that nobody is oncall (so everybody is oncall). Sounds like the gap is too wide. If these alerts are 50% "customer reported an incident" instead of a SLI, then that SRE book could spawn a cargo cult that creates work for everybody and achieves nothing because the gap is too wide. I've seen clueless managers latch onto theory that's too far ahead of the stone-age. Like "Hey guys, this month I want everybody to record their SLOs", when nobody is even recording metrics because they don't know how, or what to record. I'd do extremely boring stuff like keep a spreadsheet of incidents with columns for severity, trigger (eg release, operator error, customer load, hidden bug, system error), root cause (eg internal software bug, bad config, system resources etc). If patterns emerge, they should point to gaps in knowledge, testing, automation, monitoring etc. If there's no pattern at all, then your engineering org is in a bad way. When an engineering department is extremely dysfunctional, it needs external help. Or sometimes it needs a clueless manager / lead to quit, allowing suppressed talent too take over and fix stuff.

Normal_gaussian 4y ago

On call rota - 1 week on, n weeks off, but make sure its compensated. Primary duty of on call is triage, pushing all non-critical issues to work hours and only bringing issues they deal with through mitigation and not resolution.

evnsio 4y ago

When you say there's 2-3 devs who can actually fix issues, is that because there are only 2-3 devs, or do you have more people available but they don't know how to fix? Assuming the latter, my suggestion would be: 1) Setup a weekly on-call rotation, where one of the 3 people who can fix issues is dedicated on-call (this means the other 2 can properly switch off) 2) Find some other folks who might not be able to support immediately, but are happy to learn. 3) Configure a shadow rotation so there's always a primary responder who know what they're doing and a shadow responder who can learn on the job. 4) Make it clear the shadow person has the same expectations in terms of turning up, but their job is to ask questions, write things down, etc. I did this in a former job (and wrote about it here https://monzo.com/blog/2018/09/20/on-call https://monzo.com/blog/2018/09/20/on-call) and it worked pretty well. After a few months we had a healthy rotation of 8 on-callers, and a waitlist of shadow folks who wanted to join too. This advice obviously came from a point of near-zero context, so if you'd like to chat more I'd be happy to!

sokoloff 4y ago
Did something similar and that program turned into a milestone on many engineers’ promotion journey. (It wasn’t mandatory, but most everyone who made a certain level had successfully completed the program because you learned so much in and got such broad exposure from it.)

janci 4y ago

Define a SLA. Classify incidents by priority. Not everything must be solved immediately. Hire a developer from different time zone to cover your night time by his day time. Make your architecture more resilient. Identify most impactful improvements and stop implementation of new features until they are solved. Compensate well for the overtime. Ultimately, run away.

tomerbd 4y ago

The one who handled an error external to working hours on day x gets a day off on day x+1 otherwise you will burn out.

raffraffraff 4y ago

> So the management's idea is just that everyone have slack on their phone and whoever is available jumps in either when an alert goes off Is this a 24x7 thing? If so, this is the worst idea ever. I would quit immediately. Screw that manager. Honestly. This is clueless, and it shows a lack of care for the whole team.