11 ms·

worked in AWS for a while, but it was 5 years ago. you’re right that your manager can make or break your experience. For the first couple of years there it ki

worked in AWS for a while, but it was 5 years ago. you’re right that your manager can make or break your experience. For the first couple of years there it kinda sucked, mostly due to oncall (our ticket queue was at 3000 tickets at some point) and constantly being yelled at when things broke. We would basically only work on sev2s. Having the pager was super-stressful. We “owned” so much cruft/dead projects/experiments that a couple of time we were paged for something that we didn’t know existed. After I build a little bit of leverage / gathered some political capital I somehow ended up in the position of “team lead” with I guess management’s intent to move me to be a full time manager (after the team’s manager was PIPed). I made a good case that half the team is going to leave, me included, if we don’t do anything (6 people team). So what did we do? I introduced a “secondary” oncall. After you were oncall for a week, you were secondary for a week. While secondary you got the chance to try fixing some shit without worrying about being paged every hour. You were also motivated because you just got offcall. People went for annoyances that would generate a lot of busywork or for… fixing the alerting and monitoring (a lot of autocuts due to wrongly setup alerting thresholds or even alerts that should not have been alerts in the first place). After we exhausted the low hanging fruit, we put some effort into automating some tedious task that would take a lot of time but understood and never meant to be done manually at the scale we did them. Towards the end of the journey we aggressively deprecated/migrated the shit that was not used. By the end of this (took more than one year) we had an empty-ish oncall queue and for the first time in ages people coupd breathe (we now got a sev2 every other week - which in Amazon terms is freaking awesome). I wish this story had a happy ending. There was close to zero recognition for what happened there and most of the team migrated together, internally, to another opportunity after. I left Amazon 6 months after this migration. From what I hear from the people that stayed there, entropy took over and in another 2 years they were roughly in the same shitty place as far as oncall goes.

bpicolo 5y ago
I use reducing the rate of incoming tickets as the primary OKR for on-call engineers. Has never failed to reduce that burden over time. Just like you say - I'm convinced it's the only correct strategy to make the problem go away.
- rantwasp 5y ago
  if you set a goal but don't empower people I think overall you end up killing morale. IMHO, reducing the rate of the incoming tickets can only be done with investments that are not fire-fighting. If you're doing that I think it's awesome.
  - bpicolo 5y ago
    Yeah, absolutely - that's the whole point. If the on call expectations aren't serving that goal specifically, we need to fix/change the on call expectations, so it's a moving target over time. If there's too much firefighting, that means we owe additional support to the team or need to rebalance other work appropriately in the interim. The opposite is also true. If we don't have too much firefighting or valuable de-risking to do, we can take on more product work.
pm90 5y ago
This is a familiar story. Reducing on call load is actually generating a lot of value by freeing up developers and keeping them stress free. This fact is almost never recognized by product driven managements. Incredibly fucking annoying, considering that fixing such a system is not a trivial task. I was disappointed to hear that it ultimately didn’t work out. A quiet oncall rotation requires good processes and a somewhat competent dev team.

bpicolo 5y ago

I use reducing the rate of incoming tickets as the primary OKR for on-call engineers. Has never failed to reduce that burden over time. Just like you say - I'm convinced it's the only correct strategy to make the problem go away.

rantwasp 5y ago
if you set a goal but don't empower people I think overall you end up killing morale. IMHO, reducing the rate of the incoming tickets can only be done with investments that are not fire-fighting. If you're doing that I think it's awesome.
- bpicolo 5y ago
  Yeah, absolutely - that's the whole point. If the on call expectations aren't serving that goal specifically, we need to fix/change the on call expectations, so it's a moving target over time. If there's too much firefighting, that means we owe additional support to the team or need to rebalance other work appropriately in the interim. The opposite is also true. If we don't have too much firefighting or valuable de-risking to do, we can take on more product work.

pm90 5y ago

This is a familiar story. Reducing on call load is actually generating a lot of value by freeing up developers and keeping them stress free. This fact is almost never recognized by product driven managements. Incredibly fucking annoying, considering that fixing such a system is not a trivial task. I was disappointed to hear that it ultimately didn’t work out. A quiet oncall rotation requires good processes and a somewhat competent dev team.