4 ms·

After years of iteration, here’s what our team does. The team is remote and distributed across multiple time zones ranging from West Coast US to Western Europe

by oogali 4y ago

After years of iteration, here’s what our team does. The team is remote and distributed across multiple time zones ranging from West Coast US to Western Europe. This gets us as close to round the world coverage as we can have. There are two people on call for each shift, each shift lasts a week. It will typically (but not always) be one person from US and one person from UK/EU. This helps reduce the single personal cost and spreads it out so what might be night for one person, is morning for the other and vice versa. All of our alerts are prioritized/categorized to help prevent alert overload. For example, an alert for a test/QA environment will not fire outside of business hours, and it has a much longer time before it’s required to be ack’ed or resolved. There are two on-call rotas: critical and non-critical. Critical, production-impacting, and/or client-facing alerts are dispatched to the critical rotation. The non-critical rotation only escalates alerts during business hours, again, with a more lax timeline for acknowledgment or resolution. People are not part of both rotas at the same time. If there’s a big enough incident, the folks on call get to take off that next working day or the next one. I (the manager) am on call 24/7 for escalation. Anything that is an annoyance during on-call is a candidate for review and change. That can be anything from thresholds to code to upgrading some IaaS/SaaS subscription. Or even straight up disabling the alert if it provides no value. People can swap on-call days as they want. Typically, this happens if there’s a birthday, personal event, or PTO, and it’s worked out among team members. If no one else is available, then I’ll take their shift and act as primary.

dkyc 4y ago
How does it work for you to be on-call 24/7 for escalation? I get that that ends up happening for many committed founders/operators/managers, but I struggle how that can be a real strategy. Are you never off-grid for a bit, or drunk in a bar, or just on a real no-work vacation? There seem to be situations where being on call just isn’t feasible.
- RajT88 4y ago
  I was effectively oncall 24/7 at my job at times in 2020. I barely noticed the pandemic. I never strayed far from my computer. Also, yes, I tried not to drink much. I certainly learned what my limits are. People think I am a pretty good engineer (not amazing) but what I am known for is being able to keep that level of performance up for a long time. For my part, despite my reputation, I tried to quit a few times. Not the job, the company entirely. I have never cried at work, but came close once or twice after being up for days and unwinding from a big escalation.
  - killingtime74 4y ago
    People are always hiring, don’t sacrifice your health for a job!
  - deleted 4y ago
    [deleted]
  - wpietri 4y ago
    - bluesnowmonkey 4y ago
      I think it’s a little disrespectful to resurrect someone’s comment they tried to delete. It’s their comment, and we don’t know their reasons for deleting it.
    - remram 4y ago
      You could have posted your comment without quoting the deleted one. Let the deleted be deleted.
    - geoduck14 4y ago
      Thanks for sharing this. Burnout is real and it is a real problem. I don't think OP is entitled - that person just needs help
- moondev 4y ago
  I assume it means they are paged via escalation if the primary fails to ack which should be rare, not paged every incident.
  - oogali 4y ago
    I’m paged if the two people on call both fail to ack, which is extraordinarily rare. But what it’s there for is if the team is experiencing something that is new/novel (where I can provide some targeted guidance) or the situation is spiraling and will get worse before it gets better (where I can provide air cover).
- oogali 4y ago
  Even in all the situations you’ve listed, I still have my phone with me. If I’m going to be out of cell coverage (e.g. a plane ride, or in the countryside with spotty Internet) or simply want to be left alone, I usually plan for that in advance and do a combination of: 1) scaling back our risk exposure by rescheduling work (which requires you to have a good understanding of the business, its needs, and its timelines) and 2) shoring up the bits I feel most weary about through code, documentation, tooling, and/or contractors. The same goes for the team SMEs: reschedule where I can, crosstrain where I can’t, get headcount where none of the prior work. I’ve been on call since 1999. I had to figure out a pattern that worked for me (and my family) but wouldn’t result in a life that was boring or worse, one that I resented.
- NikolaNovak 4y ago
  (not the OP but my 100 Croatian Lipa) Depends heavily on how stable app/system is. I'm from PeopleSoft / ERP world. The same PeopleSoft HCM base application would generate one off-hours pager alert a month at one client; and half a dozen a night at another client; due to different customization/implementation/complexity of business logic and data. Any on-call/on-shift rotation system must be viewed through the lens of actual demand and need. At first client, we had 3-4 developers total who shared pagers on weekly basis, as per the OP, with no undue stress or impact on their day job. At second client, we now have multi-tier support starting with on-shift (junior but specialized ops team members who stare at computer overnight and provide immediate response), Tier 1 and Tier 2 on-call support, and multi-level escalation rotation. And yes, there are still people who get woken up all the time always, because buck eventually stops there :-/ . Being on call sucks, as per the title. I've been in 24x7 escalation roles; I don't drink to begin with so that's not an issue, but it absolutely had significant negative impact on my social & family life, sleep and stress levels. I've spent significant effort to a) Make the system better, both in terms of more reliable application, and deeper and more self-sufficient support team tree, and b) Move myself out of the role, though that relies on success in a). I do find it fascinating to occasionally meet very senior people, with family and social life, who are positively EAGER to be on-call and engaged for every little thing all the time always - and then, unfortunately, have same expectations of literally everybody else ("Let's all come on bridge, always, for everything, anytime")
  - presentation 4y ago
    Another data point from a startup, we have a few people also globally distributed on on-call, we use each other as backups in case as the commenter suggested you're drunk at a bar or camping. Our site doesn't break much (haven't had an incident for over a month now, maybe something happens once every 1-3 months and it's usually not severe) and we have redundancy in the schedule so we've come to not feel so neurotic about it.
- Aeolun 4y ago
  If this happened when I was at the bar half drunk I’d always say something to the extend of ‘do you really want me to work on the system right now?’. Sometimes the answer is yes. Sometimes the answer is no. If it’s down already it can’t get worse. To be fair, I was the first (and only) point of escalation.
  - xur17 4y ago
    > If it’s down already it can’t get worse. Oh yes it can.
gregmac 4y ago
> Anything that is an annoyance during on-call is a candidate for review and change. What does that mean in practice? Hopefully: "This woke me up last night. It's now the top priority until it's fixed so it never wakes anyone up again. Sorry product manager, your new feature will have to wait."
- oogali 4y ago
  "This woke me up last night, but it could have been something that didn't need escalation outside of business hours." "This woke me up last night and had I snoozed for 5 more minutes it could have been catastrophic, let's get some more proactive monitoring in place." "This woke me up last night and it was triggered by bad user input, we shouldn't get alerted on this but more importantly, we shouldn't allow users to submit this crap." Very rarely do I encounter alerts that are traced back to some deep architectural flaw that requires me to tango with a product manager and their roadmap. Often times, our team escalates to the engineering lead in question and a small bug fix is slipped into the next release.

by oogali 4y ago

dkyc 4y ago

How does it work for you to be on-call 24/7 for escalation? I get that that ends up happening for many committed founders/operators/managers, but I struggle how that can be a real strategy. Are you never off-grid for a bit, or drunk in a bar, or just on a real no-work vacation? There seem to be situations where being on call just isn’t feasible.

RajT88 4y ago
I was effectively oncall 24/7 at my job at times in 2020. I barely noticed the pandemic. I never strayed far from my computer. Also, yes, I tried not to drink much. I certainly learned what my limits are. People think I am a pretty good engineer (not amazing) but what I am known for is being able to keep that level of performance up for a long time. For my part, despite my reputation, I tried to quit a few times. Not the job, the company entirely. I have never cried at work, but came close once or twice after being up for days and unwinding from a big escalation.
- killingtime74 4y ago
  People are always hiring, don’t sacrifice your health for a job!
- deleted 4y ago
  [deleted]
- wpietri 4y ago
  - bluesnowmonkey 4y ago
    I think it’s a little disrespectful to resurrect someone’s comment they tried to delete. It’s their comment, and we don’t know their reasons for deleting it.
  - remram 4y ago
    You could have posted your comment without quoting the deleted one. Let the deleted be deleted.
  - geoduck14 4y ago
    Thanks for sharing this. Burnout is real and it is a real problem. I don't think OP is entitled - that person just needs help
moondev 4y ago
I assume it means they are paged via escalation if the primary fails to ack which should be rare, not paged every incident.
- oogali 4y ago
  I’m paged if the two people on call both fail to ack, which is extraordinarily rare. But what it’s there for is if the team is experiencing something that is new/novel (where I can provide some targeted guidance) or the situation is spiraling and will get worse before it gets better (where I can provide air cover).
oogali 4y ago
Even in all the situations you’ve listed, I still have my phone with me. If I’m going to be out of cell coverage (e.g. a plane ride, or in the countryside with spotty Internet) or simply want to be left alone, I usually plan for that in advance and do a combination of: 1) scaling back our risk exposure by rescheduling work (which requires you to have a good understanding of the business, its needs, and its timelines) and 2) shoring up the bits I feel most weary about through code, documentation, tooling, and/or contractors. The same goes for the team SMEs: reschedule where I can, crosstrain where I can’t, get headcount where none of the prior work. I’ve been on call since 1999. I had to figure out a pattern that worked for me (and my family) but wouldn’t result in a life that was boring or worse, one that I resented.
NikolaNovak 4y ago
(not the OP but my 100 Croatian Lipa) Depends heavily on how stable app/system is. I'm from PeopleSoft / ERP world. The same PeopleSoft HCM base application would generate one off-hours pager alert a month at one client; and half a dozen a night at another client; due to different customization/implementation/complexity of business logic and data. Any on-call/on-shift rotation system must be viewed through the lens of actual demand and need. At first client, we had 3-4 developers total who shared pagers on weekly basis, as per the OP, with no undue stress or impact on their day job. At second client, we now have multi-tier support starting with on-shift (junior but specialized ops team members who stare at computer overnight and provide immediate response), Tier 1 and Tier 2 on-call support, and multi-level escalation rotation. And yes, there are still people who get woken up all the time always, because buck eventually stops there :-/ . Being on call sucks, as per the title. I've been in 24x7 escalation roles; I don't drink to begin with so that's not an issue, but it absolutely had significant negative impact on my social & family life, sleep and stress levels. I've spent significant effort to a) Make the system better, both in terms of more reliable application, and deeper and more self-sufficient support team tree, and b) Move myself out of the role, though that relies on success in a). I do find it fascinating to occasionally meet very senior people, with family and social life, who are positively EAGER to be on-call and engaged for every little thing all the time always - and then, unfortunately, have same expectations of literally everybody else ("Let's all come on bridge, always, for everything, anytime")
- presentation 4y ago
  Another data point from a startup, we have a few people also globally distributed on on-call, we use each other as backups in case as the commenter suggested you're drunk at a bar or camping. Our site doesn't break much (haven't had an incident for over a month now, maybe something happens once every 1-3 months and it's usually not severe) and we have redundancy in the schedule so we've come to not feel so neurotic about it.
Aeolun 4y ago
If this happened when I was at the bar half drunk I’d always say something to the extend of ‘do you really want me to work on the system right now?’. Sometimes the answer is yes. Sometimes the answer is no. If it’s down already it can’t get worse. To be fair, I was the first (and only) point of escalation.
- xur17 4y ago
  > If it’s down already it can’t get worse. Oh yes it can.

gregmac 4y ago

> Anything that is an annoyance during on-call is a candidate for review and change. What does that mean in practice? Hopefully: "This woke me up last night. It's now the top priority until it's fixed so it never wakes anyone up again. Sorry product manager, your new feature will have to wait."

oogali 4y ago
"This woke me up last night, but it could have been something that didn't need escalation outside of business hours." "This woke me up last night and had I snoozed for 5 more minutes it could have been catastrophic, let's get some more proactive monitoring in place." "This woke me up last night and it was triggered by bad user input, we shouldn't get alerted on this but more importantly, we shouldn't allow users to submit this crap." Very rarely do I encounter alerts that are traced back to some deep architectural flaw that requires me to tango with a product manager and their roadmap. Often times, our team escalates to the engineering lead in question and a small bug fix is slipped into the next release.