7 ms·

How does it work for you to be on-call 24/7 for escalation? I get that that ends up happening for many committed founders/operators/managers, but I struggle how

by dkyc 4y ago

How does it work for you to be on-call 24/7 for escalation? I get that that ends up happening for many committed founders/operators/managers, but I struggle how that can be a real strategy. Are you never off-grid for a bit, or drunk in a bar, or just on a real no-work vacation? There seem to be situations where being on call just isn’t feasible.

RajT88 4y ago
I was effectively oncall 24/7 at my job at times in 2020. I barely noticed the pandemic. I never strayed far from my computer. Also, yes, I tried not to drink much. I certainly learned what my limits are. People think I am a pretty good engineer (not amazing) but what I am known for is being able to keep that level of performance up for a long time. For my part, despite my reputation, I tried to quit a few times. Not the job, the company entirely. I have never cried at work, but came close once or twice after being up for days and unwinding from a big escalation.
- killingtime74 4y ago
  People are always hiring, don’t sacrifice your health for a job!
- deleted 4y ago
  [deleted]
- wpietri 4y ago
  - bluesnowmonkey 4y ago
    I think it’s a little disrespectful to resurrect someone’s comment they tried to delete. It’s their comment, and we don’t know their reasons for deleting it.
  - remram 4y ago
    You could have posted your comment without quoting the deleted one. Let the deleted be deleted.
  - geoduck14 4y ago
    Thanks for sharing this. Burnout is real and it is a real problem. I don't think OP is entitled - that person just needs help
moondev 4y ago
I assume it means they are paged via escalation if the primary fails to ack which should be rare, not paged every incident.
- oogali 4y ago
  I’m paged if the two people on call both fail to ack, which is extraordinarily rare. But what it’s there for is if the team is experiencing something that is new/novel (where I can provide some targeted guidance) or the situation is spiraling and will get worse before it gets better (where I can provide air cover).
oogali 4y ago
Even in all the situations you’ve listed, I still have my phone with me. If I’m going to be out of cell coverage (e.g. a plane ride, or in the countryside with spotty Internet) or simply want to be left alone, I usually plan for that in advance and do a combination of: 1) scaling back our risk exposure by rescheduling work (which requires you to have a good understanding of the business, its needs, and its timelines) and 2) shoring up the bits I feel most weary about through code, documentation, tooling, and/or contractors. The same goes for the team SMEs: reschedule where I can, crosstrain where I can’t, get headcount where none of the prior work. I’ve been on call since 1999. I had to figure out a pattern that worked for me (and my family) but wouldn’t result in a life that was boring or worse, one that I resented.
NikolaNovak 4y ago
(not the OP but my 100 Croatian Lipa) Depends heavily on how stable app/system is. I'm from PeopleSoft / ERP world. The same PeopleSoft HCM base application would generate one off-hours pager alert a month at one client; and half a dozen a night at another client; due to different customization/implementation/complexity of business logic and data. Any on-call/on-shift rotation system must be viewed through the lens of actual demand and need. At first client, we had 3-4 developers total who shared pagers on weekly basis, as per the OP, with no undue stress or impact on their day job. At second client, we now have multi-tier support starting with on-shift (junior but specialized ops team members who stare at computer overnight and provide immediate response), Tier 1 and Tier 2 on-call support, and multi-level escalation rotation. And yes, there are still people who get woken up all the time always, because buck eventually stops there :-/ . Being on call sucks, as per the title. I've been in 24x7 escalation roles; I don't drink to begin with so that's not an issue, but it absolutely had significant negative impact on my social & family life, sleep and stress levels. I've spent significant effort to a) Make the system better, both in terms of more reliable application, and deeper and more self-sufficient support team tree, and b) Move myself out of the role, though that relies on success in a). I do find it fascinating to occasionally meet very senior people, with family and social life, who are positively EAGER to be on-call and engaged for every little thing all the time always - and then, unfortunately, have same expectations of literally everybody else ("Let's all come on bridge, always, for everything, anytime")
- presentation 4y ago
  Another data point from a startup, we have a few people also globally distributed on on-call, we use each other as backups in case as the commenter suggested you're drunk at a bar or camping. Our site doesn't break much (haven't had an incident for over a month now, maybe something happens once every 1-3 months and it's usually not severe) and we have redundancy in the schedule so we've come to not feel so neurotic about it.
Aeolun 4y ago
If this happened when I was at the bar half drunk I’d always say something to the extend of ‘do you really want me to work on the system right now?’. Sometimes the answer is yes. Sometimes the answer is no. If it’s down already it can’t get worse. To be fair, I was the first (and only) point of escalation.
- xur17 4y ago
  > If it’s down already it can’t get worse. Oh yes it can.

by dkyc 4y ago

RajT88 4y ago

I was effectively oncall 24/7 at my job at times in 2020. I barely noticed the pandemic. I never strayed far from my computer. Also, yes, I tried not to drink much. I certainly learned what my limits are. People think I am a pretty good engineer (not amazing) but what I am known for is being able to keep that level of performance up for a long time. For my part, despite my reputation, I tried to quit a few times. Not the job, the company entirely. I have never cried at work, but came close once or twice after being up for days and unwinding from a big escalation.

killingtime74 4y ago
People are always hiring, don’t sacrifice your health for a job!
deleted 4y ago
[deleted]
wpietri 4y ago
- bluesnowmonkey 4y ago
  I think it’s a little disrespectful to resurrect someone’s comment they tried to delete. It’s their comment, and we don’t know their reasons for deleting it.
- remram 4y ago
  You could have posted your comment without quoting the deleted one. Let the deleted be deleted.
- geoduck14 4y ago
  Thanks for sharing this. Burnout is real and it is a real problem. I don't think OP is entitled - that person just needs help

moondev 4y ago

I assume it means they are paged via escalation if the primary fails to ack which should be rare, not paged every incident.

oogali 4y ago
I’m paged if the two people on call both fail to ack, which is extraordinarily rare. But what it’s there for is if the team is experiencing something that is new/novel (where I can provide some targeted guidance) or the situation is spiraling and will get worse before it gets better (where I can provide air cover).

oogali 4y ago

Even in all the situations you’ve listed, I still have my phone with me. If I’m going to be out of cell coverage (e.g. a plane ride, or in the countryside with spotty Internet) or simply want to be left alone, I usually plan for that in advance and do a combination of: 1) scaling back our risk exposure by rescheduling work (which requires you to have a good understanding of the business, its needs, and its timelines) and 2) shoring up the bits I feel most weary about through code, documentation, tooling, and/or contractors. The same goes for the team SMEs: reschedule where I can, crosstrain where I can’t, get headcount where none of the prior work. I’ve been on call since 1999. I had to figure out a pattern that worked for me (and my family) but wouldn’t result in a life that was boring or worse, one that I resented.

NikolaNovak 4y ago

(not the OP but my 100 Croatian Lipa) Depends heavily on how stable app/system is. I'm from PeopleSoft / ERP world. The same PeopleSoft HCM base application would generate one off-hours pager alert a month at one client; and half a dozen a night at another client; due to different customization/implementation/complexity of business logic and data. Any on-call/on-shift rotation system must be viewed through the lens of actual demand and need. At first client, we had 3-4 developers total who shared pagers on weekly basis, as per the OP, with no undue stress or impact on their day job. At second client, we now have multi-tier support starting with on-shift (junior but specialized ops team members who stare at computer overnight and provide immediate response), Tier 1 and Tier 2 on-call support, and multi-level escalation rotation. And yes, there are still people who get woken up all the time always, because buck eventually stops there :-/ . Being on call sucks, as per the title. I've been in 24x7 escalation roles; I don't drink to begin with so that's not an issue, but it absolutely had significant negative impact on my social & family life, sleep and stress levels. I've spent significant effort to a) Make the system better, both in terms of more reliable application, and deeper and more self-sufficient support team tree, and b) Move myself out of the role, though that relies on success in a). I do find it fascinating to occasionally meet very senior people, with family and social life, who are positively EAGER to be on-call and engaged for every little thing all the time always - and then, unfortunately, have same expectations of literally everybody else ("Let's all come on bridge, always, for everything, anytime")

presentation 4y ago
Another data point from a startup, we have a few people also globally distributed on on-call, we use each other as backups in case as the commenter suggested you're drunk at a bar or camping. Our site doesn't break much (haven't had an incident for over a month now, maybe something happens once every 1-3 months and it's usually not severe) and we have redundancy in the schedule so we've come to not feel so neurotic about it.

Aeolun 4y ago

If this happened when I was at the bar half drunk I’d always say something to the extend of ‘do you really want me to work on the system right now?’. Sometimes the answer is yes. Sometimes the answer is no. If it’s down already it can’t get worse. To be fair, I was the first (and only) point of escalation.

xur17 4y ago
> If it’s down already it can’t get worse. Oh yes it can.