5 ms·

Do you have kids? hobbies? friends? Do you like to go fishing on weekends? Do you like to go out on a date on weeknights? Do you like going to visit family on

Do you have kids? hobbies? friends? Do you like to go fishing on weekends? Do you like to go out on a date on weeknights? Do you like going to visit family on Christmas and Thanksgiving and leaving your laptop at home? It's great explaining to your 11 year old that you can't go to their recital because your on call isn't it? It boggles my mind that you suggest people lose skills by not using them often enough... these on call rotations aren't skills. What's weird, is that we default to look at the goodness or badness of something purely on "skills" and "money". Yes.. yes.. we've all read the SRE book and understand Googles approach to flexibility - but... it doesn't have to be that way. Ask yourself.. what's next? What are you looking to do after 25 years of being on call? Will you still enjoy it? I'll never have those nights and weekends back that I spent being on call for products and services that don't exist anymore. Sure, I made some money - but it didn't have to be at such a great cost.

vitus 5y ago
At least at Google, oncall shifts aren't set in stone. If you need to swap with a coworker (even for a few hours), you can. Devs should not be holding 15-minute (much less 5-minute) pagers; if your system is really that critical, you should be working to onboard an SRE team. > these on call rotations aren't skills. Troubleshooting live production issues is a skill. Triaging incidents to prioritize actions to minimize user harm is a skill. Assessing tradeoffs of mitigation strategies is a skill. The very tangible scenario of being woken up in the middle of the night builds empathy with your devops / SRE organization (which can manifest itself in designing more resilient systems, engaging in less risky behavior, improving documentation for the next oncaller, etc). Oncall isn't for everyone, and it's certainly not worthwhile if you don't take away any lessons from it. But devs are ultimately the closest ones to their code, and they should take advantage of the opportunity to understand how it behaves in production. > Ask yourself.. what's next? What are you looking to do after 25 years of being on call? Will you still enjoy it? I've had bad weeks of oncall as well as quiet weeks. If it ever gets to a point where bad weeks are much more frequent, and nobody's doing anything about it, then yeah, I won't want to do it anymore. But as long as it's still providing value beyond humans providing blood / sweat / tears, and as long as we're actually trying to make things better, I don't mind holding a pager a few weeks a year.
- supernovae 5y ago
  I can troubleshoot during business hours. I can triag during business hours. I can asset tradeoffs and mitigation strategies during business hours. Being woken up in the middle of the night never builds empathy and if you pay enough attention to the industry, we're leadning more towards trauma, depression, anxiety and self-diagnosing ADHD. BTW, me being anti-on call is not anti-resiliency... in fact, I don't think on-call is resilient at all with how most organizations operate since the resilience isn't adaptive capacity, but "do your day job and be on call" - no matter HOW much we talk about "your on call week is the week your on call and you focus on that" - it never happens. Never ever.
  - vitus 5y ago
    I get that any one of us can do all of these things during normal business hours (assuming that said situations arise then). But "can" isn't the same as "will". There are plenty of things I'll never get around to if I can't explicitly set aside time for oncall. > Being woken up in the middle of the night never builds empathy and if you pay enough attention to the industry, we're leadning more towards trauma, depression, anxiety and self-diagnosing ADHD. Being woken up in the middle of the night leaves me grumpy and motivates me the following work day to figure out how I can avoid getting woken up again for the same issue. This is somewhat like dogfooding the oncall experience. If I'm trying to troubleshoot something but the logs are unhelpful or the runbook is too vague for my sleep-deprived brain to figure out, I can fix that for next time. If it takes me half an hour to answer a simple question, it's worth considering how to improve our monitoring and dashboards. If I can't follow my own documented process, how can I expect a team who didn't build these systems to do so? As I said, I don't think oncall is for everyone. Oncall as a hidden job requirement is especially problematic, especially if your team / company doesn't expect (and/or mandate!) that your regular project work will be on the back burner during your shifts. Ideally oncall should be opt-in, but there's a balancing act with not having enough people to avoid rapid burnout. Google limits how much you can earn per oncall compensation (for SWEs, 2 weeks per quarter) -- if you're regularly exceeding that limit, then your oncall rotation isn't healthy. There's a lot of horror stories in these comments about oncall rotations, and quite frankly I wouldn't want to join one of those either. ... Another program Google has that's like oncall on steroids: Mission Control [0]. It's an exchange program where SWEs embed themselves in SRE teams, up to and including taking on their pager. It's a fantastic opportunity for gaining a deeper understanding / appreciation of production, but it can potentially be an even greater imposition. If you're an unattached 20-something, you might welcome the opportunity to live in Sydney for 6 months, but if you've got kids, that's not reasonable. (Of course, with the pandemic, relocation to a different country isn't really a thing at this time.) [0] https://cloud.google.com/blog/products/gcp/adventures-in-sre-land-welcome-to-google-mission-control https://cloud.google.com/blog/products/gcp/adventures-in-sre...
joshuamorton 5y ago
I'm confused, the post you're replying to says that the Google model means you don't have nights OnCall. > . these on call rotations aren't skills. Incident response/management is absolutely a skill.
- supernovae 5y ago
  You don't have to be on call to have that skill. BTW, the google model works for them... i'm being more in general since the issue was facebook vs google vs everyone else that I was replying to.
  - joshuamorton 5y ago
    Incident response requires an incident responder. That is, a person who is alerted when things break. That person (or persons) is defacto on call. If you are never responsible for responding to incidents, your ability to respond to incidents will atrophy.
    - supernovae 5y ago
      I would advise such a person works during their normal 8-5 and that "on call" is just a form of work intake for a normal job - not an exception that happens after hours or on weekends when most people refer to "on call".
      - joshuamorton 5y ago
        Ok, but then what do you do on weekends? Like you still need people, who have the skills to do the oncall work, to man the pager during off-work hours. Hiring a weekend person doesn't work (they don't have the skills since they're not really part of the team, + bus factor). Forcing people to work weekends doesn't work. Some sort of rotation seems the least worst option.
- mathteddybear 5y ago
  According to that post, "You don't have nights OnCall" applies only to SREs covering the systems of the highest profile. Which is not even all SREs Anyways SREs are software developers as well. The distinction SWE/SRE came up to be in times of SOX compliance; nowadays there are also some oncall-related distinctions (that SREs are more likely to be in such multiple-timezone-office teams, for instance, could be true).
cletus 5y ago
> Do you have kids? hobbies? friends? Do you like to go fishing on weekends? Do you like having a job that gives you money to pay for all those things as well as the basic ability to, you know, live? How far do you want to take this? Is working 40 hours a week taking time away from your children, friends or hobbies? Of course it is. But nobody owes you a living. A job is you trading your time and your skills to someone else in a way that provides that person value. There are going to be parts of that job you like and probably parts you don't. If you really want to avoid oncall then fine, do it. That will likely limit your career opportunities because some jobs will require it or your coworkers may be rewarded (monetarily and careerwise) because they're doing something you're not. There's nothing wrong with that choice but at the same time you don't get to complain because others benefit from doing something you won't. > but... it doesn't have to be that way. For 24/7 production services then who exactly supports those services? > I'll never have those nights and weekends back that I spent being on call for products and services that don't exist anymore. There's a pretty good chance that nothing you worked on will exist in 25 years. What's your point?
- PeterisP 5y ago
  The issue seems to be about unreasonable work hours. On-call time is work time - perhaps not "full" work time, but definitely not free time or rest time. If you're on call for 24 hours of Saturday, that's fine, but then work is taking up 24 hours more than your usual week, and deserves a major change in compensation or extra rest time (e.g. the following monday off) as it's a big change from a 40 hour or 50 hour week. If your team has allocated every engineer to an on-call day every week, that's the equivalent of expecting all of your engineers to devote 60+ hour weeks every week to their job - and IMHO that's not okay; some of your engineers may be fine with that (and would deserve what essentially is overtime compensation), but having that as a requirement for everyone is not reasonable in non-toxic workplaces, if there's no good place in your team for people who will only work 40-ish hours, then that would be pretty much discriminatory on multiple groups. Having a "week of on-call" is equivalent to a continuous 168-hour shift, which is simply not done anywhere else and should not be a thing. If you want 24/7 production services, then you need to map shifts of support. For continuous work you just run 3x8 or 2x12 hour shifts; but outside of IT industry where you need "spotty monitoring", 24 hours on / 48 hours off is a commonly used system - but it's not a 24 hour shift on top of another full time job.
achiang 5y ago
Google SRE here. Point of clarification on "losing skills" by not being oncall enough. Google designs its SRE teams to scale sublinearly to the service, which means we're often responsible for entire portfolios, not just a single service. It's common for individuals to be SMEs on a subset of the portfolio, as the focus of their coding projects. Obviously the rest of the portfolio is also undergoing continuous change, and so to remain a broadly effective oncaller across the entire portfolio, you need regular production exposure to the parts you interact with less frequently. Those are the specific bits that get rusty with disuse.

vitus 5y ago

At least at Google, oncall shifts aren't set in stone. If you need to swap with a coworker (even for a few hours), you can. Devs should not be holding 15-minute (much less 5-minute) pagers; if your system is really that critical, you should be working to onboard an SRE team. > these on call rotations aren't skills. Troubleshooting live production issues is a skill. Triaging incidents to prioritize actions to minimize user harm is a skill. Assessing tradeoffs of mitigation strategies is a skill. The very tangible scenario of being woken up in the middle of the night builds empathy with your devops / SRE organization (which can manifest itself in designing more resilient systems, engaging in less risky behavior, improving documentation for the next oncaller, etc). Oncall isn't for everyone, and it's certainly not worthwhile if you don't take away any lessons from it. But devs are ultimately the closest ones to their code, and they should take advantage of the opportunity to understand how it behaves in production. > Ask yourself.. what's next? What are you looking to do after 25 years of being on call? Will you still enjoy it? I've had bad weeks of oncall as well as quiet weeks. If it ever gets to a point where bad weeks are much more frequent, and nobody's doing anything about it, then yeah, I won't want to do it anymore. But as long as it's still providing value beyond humans providing blood / sweat / tears, and as long as we're actually trying to make things better, I don't mind holding a pager a few weeks a year.

supernovae 5y ago
I can troubleshoot during business hours. I can triag during business hours. I can asset tradeoffs and mitigation strategies during business hours. Being woken up in the middle of the night never builds empathy and if you pay enough attention to the industry, we're leadning more towards trauma, depression, anxiety and self-diagnosing ADHD. BTW, me being anti-on call is not anti-resiliency... in fact, I don't think on-call is resilient at all with how most organizations operate since the resilience isn't adaptive capacity, but "do your day job and be on call" - no matter HOW much we talk about "your on call week is the week your on call and you focus on that" - it never happens. Never ever.
- vitus 5y ago
  I get that any one of us can do all of these things during normal business hours (assuming that said situations arise then). But "can" isn't the same as "will". There are plenty of things I'll never get around to if I can't explicitly set aside time for oncall. > Being woken up in the middle of the night never builds empathy and if you pay enough attention to the industry, we're leadning more towards trauma, depression, anxiety and self-diagnosing ADHD. Being woken up in the middle of the night leaves me grumpy and motivates me the following work day to figure out how I can avoid getting woken up again for the same issue. This is somewhat like dogfooding the oncall experience. If I'm trying to troubleshoot something but the logs are unhelpful or the runbook is too vague for my sleep-deprived brain to figure out, I can fix that for next time. If it takes me half an hour to answer a simple question, it's worth considering how to improve our monitoring and dashboards. If I can't follow my own documented process, how can I expect a team who didn't build these systems to do so? As I said, I don't think oncall is for everyone. Oncall as a hidden job requirement is especially problematic, especially if your team / company doesn't expect (and/or mandate!) that your regular project work will be on the back burner during your shifts. Ideally oncall should be opt-in, but there's a balancing act with not having enough people to avoid rapid burnout. Google limits how much you can earn per oncall compensation (for SWEs, 2 weeks per quarter) -- if you're regularly exceeding that limit, then your oncall rotation isn't healthy. There's a lot of horror stories in these comments about oncall rotations, and quite frankly I wouldn't want to join one of those either. ... Another program Google has that's like oncall on steroids: Mission Control [0]. It's an exchange program where SWEs embed themselves in SRE teams, up to and including taking on their pager. It's a fantastic opportunity for gaining a deeper understanding / appreciation of production, but it can potentially be an even greater imposition. If you're an unattached 20-something, you might welcome the opportunity to live in Sydney for 6 months, but if you've got kids, that's not reasonable. (Of course, with the pandemic, relocation to a different country isn't really a thing at this time.) [0] https://cloud.google.com/blog/products/gcp/adventures-in-sre-land-welcome-to-google-mission-control https://cloud.google.com/blog/products/gcp/adventures-in-sre...

joshuamorton 5y ago

I'm confused, the post you're replying to says that the Google model means you don't have nights OnCall. > . these on call rotations aren't skills. Incident response/management is absolutely a skill.

supernovae 5y ago
You don't have to be on call to have that skill. BTW, the google model works for them... i'm being more in general since the issue was facebook vs google vs everyone else that I was replying to.
- joshuamorton 5y ago
  Incident response requires an incident responder. That is, a person who is alerted when things break. That person (or persons) is defacto on call. If you are never responsible for responding to incidents, your ability to respond to incidents will atrophy.
  - supernovae 5y ago
    I would advise such a person works during their normal 8-5 and that "on call" is just a form of work intake for a normal job - not an exception that happens after hours or on weekends when most people refer to "on call".
    - joshuamorton 5y ago
      Ok, but then what do you do on weekends? Like you still need people, who have the skills to do the oncall work, to man the pager during off-work hours. Hiring a weekend person doesn't work (they don't have the skills since they're not really part of the team, + bus factor). Forcing people to work weekends doesn't work. Some sort of rotation seems the least worst option.
mathteddybear 5y ago
According to that post, "You don't have nights OnCall" applies only to SREs covering the systems of the highest profile. Which is not even all SREs Anyways SREs are software developers as well. The distinction SWE/SRE came up to be in times of SOX compliance; nowadays there are also some oncall-related distinctions (that SREs are more likely to be in such multiple-timezone-office teams, for instance, could be true).

cletus 5y ago

> Do you have kids? hobbies? friends? Do you like to go fishing on weekends? Do you like having a job that gives you money to pay for all those things as well as the basic ability to, you know, live? How far do you want to take this? Is working 40 hours a week taking time away from your children, friends or hobbies? Of course it is. But nobody owes you a living. A job is you trading your time and your skills to someone else in a way that provides that person value. There are going to be parts of that job you like and probably parts you don't. If you really want to avoid oncall then fine, do it. That will likely limit your career opportunities because some jobs will require it or your coworkers may be rewarded (monetarily and careerwise) because they're doing something you're not. There's nothing wrong with that choice but at the same time you don't get to complain because others benefit from doing something you won't. > but... it doesn't have to be that way. For 24/7 production services then who exactly supports those services? > I'll never have those nights and weekends back that I spent being on call for products and services that don't exist anymore. There's a pretty good chance that nothing you worked on will exist in 25 years. What's your point?

PeterisP 5y ago
The issue seems to be about unreasonable work hours. On-call time is work time - perhaps not "full" work time, but definitely not free time or rest time. If you're on call for 24 hours of Saturday, that's fine, but then work is taking up 24 hours more than your usual week, and deserves a major change in compensation or extra rest time (e.g. the following monday off) as it's a big change from a 40 hour or 50 hour week. If your team has allocated every engineer to an on-call day every week, that's the equivalent of expecting all of your engineers to devote 60+ hour weeks every week to their job - and IMHO that's not okay; some of your engineers may be fine with that (and would deserve what essentially is overtime compensation), but having that as a requirement for everyone is not reasonable in non-toxic workplaces, if there's no good place in your team for people who will only work 40-ish hours, then that would be pretty much discriminatory on multiple groups. Having a "week of on-call" is equivalent to a continuous 168-hour shift, which is simply not done anywhere else and should not be a thing. If you want 24/7 production services, then you need to map shifts of support. For continuous work you just run 3x8 or 2x12 hour shifts; but outside of IT industry where you need "spotty monitoring", 24 hours on / 48 hours off is a commonly used system - but it's not a 24 hour shift on top of another full time job.

achiang 5y ago

Google SRE here. Point of clarification on "losing skills" by not being oncall enough. Google designs its SRE teams to scale sublinearly to the service, which means we're often responsible for entire portfolios, not just a single service. It's common for individuals to be SMEs on a subset of the portfolio, as the focus of their coding projects. Obviously the rest of the portfolio is also undergoing continuous change, and so to remain a broadly effective oncaller across the entire portfolio, you need regular production exposure to the parts you interact with less frequently. Those are the specific bits that get rusty with disuse.