6 ms·

My problem is that I enjoy many aspects of SRE work, but I absolutely despise being on call. I've since transitioned into onto a different career track, but I

My problem is that I enjoy many aspects of SRE work, but I absolutely despise being on call. I've since transitioned into onto a different career track, but I have long wished to find some way to use my combination of unix sysadmin and software engineering skills without ever having to be on call. In the companies I worked in (including Google), I never really found that.

deleted 10y ago
[deleted]
general_ai 10y ago
Even SWEs are oncall. Not as much as SREs, but it's really unavoidable. Someone needs to be there if shit hits the fan.
- zippergz 10y ago
  Depends on the project. As a swe I got paged zero times.
  - general_ai 10y ago
    For anything that actually receives significant traffic, runs in a lot of cells, and is under active development, Telebot will call you pretty regularly when you're oncall. But hey, congrats for finding a place you like, that's exactly how things are supposed to work there.
  - twright0 10y ago
    You can be on call without ever getting paged. As long as the expectation exists that you're available, it doesn't matter whether you actually get paged or not - you're already shifting your plans so you stay within reach of a laptop, with cell signal, etc. Being paged is the least bad part of being on call. The restrictions on your life outside of work are what really grate.
    - CydeWeys 10y ago
      I disagree. Being paged in the middle of the night is the worst part of being on call, especially if it's a bad incident that takes hours to resolve. That can ruin your entire next day. Having to bring my laptop with me to social events isn't nearly as bad. It's rare that being on-call affects my plans.
      - user5994461 10y ago
        You're both right. If you are on call one week every 3 months, it's mostly the calls that are annoying, especially at night. If you are on call more often (e.g. 1/2 weekend for small companies/teams), it stops mattering how often pages happen. You are mentally on-call all the time and are forced to be physically ready. It's a huge drain on your life.
        CydeWeys 10y ago
        I'm on call a day at a time roughly every nine days or so. It definitely matters how often pages happen. On my team they are very infrequent, so being oncall is hardly a burden at all. I just have to remember to bring my laptop. Maybe it's a psychological thing, with different people responding differently. But it isn't a huge drain on my life, at all.
        TheGrumpyBrit 10y ago
        It depends on the arrangement too. If you're on a flat rate for being on call, being called out is the worst part. If you only get paid if you get called, being called out kind of makes it worth changing your plans. I get a small amount for each day I'm on call, plus an extra payment if I'm actually called. That seems a fair balance - I'm compensated reasonably for making myself available, and the actual callout payment makes it worth getting up at 3am a couple of times a year.
    - zippergz 10y ago
      For me, both were terrible. The restrictions were a huge pain in the ass, and annoying to my wife. But I'm REALLY not at my best if I don't get a good night's sleep, so every night that I got paged during the night basically ruined me for the next day.
  - aczerepinski 10y ago
    I used to rotate on-call in a different industry, and I absolutely hated it. I'd go to a party at a friend's house, and would have to immediately get on their wifi network + secure a quiet room where I could go if the phone rang. Couldn't take weekend trips to go hiking or kayaking. My wife felt like she was on-call too, because it impacted what we as a family could do with our free time. The calls themselves were no big deal.
    - nickjj 10y ago
      I'm surprised so many people agree to be on call. It's basically you letting a company hire you for 24 hours while being paid 9-5. I was never on call but I have a friend who was. Like you mentioned, it plays a huge role in how you live your life. That lingering feeling of they can call you any second haunts you day in and day out. Even going to the movies must be a terrible experience? You either feel guilty for turning your pager off, or you spend the movie just ruminating about how it will end early for you if you get called. How do you even sleep at night knowing that someone might call you any second? Sounds worse than prison honestly.
      - daenney 10y ago
        > I'm surprised so many people agree to be on call. It's basically you letting a company hire you for 24 hours while being paid 9-5. Sounds like you need to renegotiate that. When I'm on call I don't get paid 9-5. There's a rather nice on-call bonus which pretty much means I get paid the full 24hrs. Might also depend on the country, some have laws around these kinds of things. For example, I can only be on-call for a 1 week stretch and have to be given a (paid) day off after. I can also not be on-call again for another couple of days. > I was never on call but I have a friend who was. Like you mentioned, it plays a huge role in how you live your life. That lingering feeling of they can call you any second haunts you day in and day out. If being on-call gives you this lingering feeling of doom then don't be on-call. If your systems are in that much of a fucked up state then you push back until that's sorted. Sometimes you need to let stuff burn. Being on-call does not have to equal being in a constant state of stress on the verge of panic attacks. > Even going to the movies must be a terrible experience? You either feel guilty for turning your pager off, or you spend the movie just ruminating about how it will end early for you if you get called. I don't go to the movies when I'm on-call. Turning your pager off at that point is irresponsible and not what I'm being paid for. In most companies you're only on-call for short periods at a time, it doesn't take over your life. You just go to the movies the day after, just like you'd plan any other activity and overlapping commitment. > How do you even sleep at night knowing that someone might call you any second? Sounds worse than prison honestly. By using Quiet Hours on your phone correctly and configuring the numbers that pages can come from to always go through. You get to bed and you sleep. The first few times you might sleep a bit less over it. Eventually you get the hang of it. A big part in this is knowing that you don't get paged for random crap but only ever when there's something truly wrong that can't wait until the next morning. Correctly behaving systems and not too trigger-happy alerting are crucial to this.
- falcolas 10y ago
  This can be done without having to call people in the middle of the night, by having distributed teams. When a company has people working regular shifts for all 24 hours, nobody is having to be woken up; someone is just working. I've lived this as a DBA, and it made so much more sense than expecting someone to be coherent and make the right decisions at 3am. The choice to put people to be on call is an explicitly sacrificing reliability for the sake of a budget. At what point is reliability worth more than the cost of a few remote employees?
  - rincebrain 10y ago
    That's still an oncall rotation, just not outside of "work hours".
    - unethical_ban 10y ago
      When people complain about oncall, the off-hours portion is almost guaranteed to be the part complained about. Furthermore, I'd argue that if there is a dedicated group of people whose job it is SOLELY to work outages and help tickets, and their job is 100% that, it's not oncall. Oncall is "hey, you have these projects to do, but every 6 weeks you have to also answer every IM/phonecall/ticket escalation as well".
      - yebyen 10y ago
        Yes, I worked at a company where tech department was severely under-manned, and we had to be great at "SRE" type stuff. (I will definitely be reading this book, I even applied for this SRE position at Google while I was still working there...) The number one problem was that (starting in Year 4) there was nobody else with the knowledge to be on-call. So, I was on-call 24/7-365 for 5 years. Realistically it was a small enough company with localized (mostly east-coast USA) clients, that things going wrong in the middle of the night was pretty unusual. I had a great boss who mentored me to be able to handle these things by myself, and he mostly put the right tools in my hands so that it was possible to avoid getting woke in the middle of the night because anything went wrong short of a disk going bad. It happened often enough to be a good reason to want to leave! It was my first serious job, and for all its faults it was a pretty great job until my boss got a better offer. When he left, it seemed that there was no choice but to implement vSAN and VSphere. And then I actually didn't need to be woke by disks going bad in the middle of the night anymore. When I did go, there was nobody prepared to handle 75% of the incidents I would have been needed to work on. (I assume that many of those things just stay broken now when they go wrong.)
StreamBright 10y ago
You can be oncall from home. I found it perfect, just hanging around writing some code and get occasionally a page or two about somebody misunderstanding what production readiness means. :) On a more serious note, it is fun to work as an SRE, lots of problems you haven't seen before, many opportunities to learn about large scale systems. This sort of knowledge and view point that comes with it is invaluable for other companies too, you can move forward with your career faster after being an SRE for few years. (Amazon asks you do that for a year before you can move on to a different role).
- szopa 10y ago
  Yeah, being oncall for something like a well behaved backend system may be quite nice. I was an SRE for YouTube, and it was an almost constant bloodbath. YouTube's code changes at a relatively fast pace, there's a ton of developers, and it depends on a ton of different backends, and a problem with any of them would make us suffer. To make it worse I had a bit of bad luck, and was a magnet for weird and unlikely outages (bracing for my shift was a running joke in the team). So, if I was oncall at home it usually meant that shit started happening before breakfast and I didn't get enough quiet time to take my 10 minute bus ride to the office :) This said, I really enjoyed the experience. Yes, it was tiring and stressful, but it was also super interesting and exciting. Being responsible for such a huge site was incredible, and the feeling of figuring out how to overcome a big outage was exhilarating. I actually miss the pager drama from time to time (my wife does not).
  - thewhitetulip 10y ago
    When you say outage does it mean youtube.com doesn't work at all, like reddit does some times or just few nodes of your load balancer give out?
- sateesh 10y ago
  I agree being an SRE exposes one to kind of problems that can occur in large scale production systems, the problems which one might not have thought during development or not unearthed during QA. But I think being oncall is flipside of being a SRE. To me being Oncall is oncall, be it be from home or oncall from office. If you are oncall during non-working hours, except what you save for the commute to being on office, your focus is drawn away from what you are doing (might be spending time with family, or book/movie you are enjoying, sleeping, or working on your project).

deleted 10y ago

[deleted]

general_ai 10y ago

Even SWEs are oncall. Not as much as SREs, but it's really unavoidable. Someone needs to be there if shit hits the fan.

zippergz 10y ago
Depends on the project. As a swe I got paged zero times.
- general_ai 10y ago
  For anything that actually receives significant traffic, runs in a lot of cells, and is under active development, Telebot will call you pretty regularly when you're oncall. But hey, congrats for finding a place you like, that's exactly how things are supposed to work there.
- twright0 10y ago
  You can be on call without ever getting paged. As long as the expectation exists that you're available, it doesn't matter whether you actually get paged or not - you're already shifting your plans so you stay within reach of a laptop, with cell signal, etc. Being paged is the least bad part of being on call. The restrictions on your life outside of work are what really grate.
  - CydeWeys 10y ago
    I disagree. Being paged in the middle of the night is the worst part of being on call, especially if it's a bad incident that takes hours to resolve. That can ruin your entire next day. Having to bring my laptop with me to social events isn't nearly as bad. It's rare that being on-call affects my plans.
    - user5994461 10y ago
      You're both right. If you are on call one week every 3 months, it's mostly the calls that are annoying, especially at night. If you are on call more often (e.g. 1/2 weekend for small companies/teams), it stops mattering how often pages happen. You are mentally on-call all the time and are forced to be physically ready. It's a huge drain on your life.
      - CydeWeys 10y ago
        I'm on call a day at a time roughly every nine days or so. It definitely matters how often pages happen. On my team they are very infrequent, so being oncall is hardly a burden at all. I just have to remember to bring my laptop. Maybe it's a psychological thing, with different people responding differently. But it isn't a huge drain on my life, at all.
      - TheGrumpyBrit 10y ago
        It depends on the arrangement too. If you're on a flat rate for being on call, being called out is the worst part. If you only get paid if you get called, being called out kind of makes it worth changing your plans. I get a small amount for each day I'm on call, plus an extra payment if I'm actually called. That seems a fair balance - I'm compensated reasonably for making myself available, and the actual callout payment makes it worth getting up at 3am a couple of times a year.
  - zippergz 10y ago
    For me, both were terrible. The restrictions were a huge pain in the ass, and annoying to my wife. But I'm REALLY not at my best if I don't get a good night's sleep, so every night that I got paged during the night basically ruined me for the next day.
- aczerepinski 10y ago
  I used to rotate on-call in a different industry, and I absolutely hated it. I'd go to a party at a friend's house, and would have to immediately get on their wifi network + secure a quiet room where I could go if the phone rang. Couldn't take weekend trips to go hiking or kayaking. My wife felt like she was on-call too, because it impacted what we as a family could do with our free time. The calls themselves were no big deal.
  - nickjj 10y ago
    I'm surprised so many people agree to be on call. It's basically you letting a company hire you for 24 hours while being paid 9-5. I was never on call but I have a friend who was. Like you mentioned, it plays a huge role in how you live your life. That lingering feeling of they can call you any second haunts you day in and day out. Even going to the movies must be a terrible experience? You either feel guilty for turning your pager off, or you spend the movie just ruminating about how it will end early for you if you get called. How do you even sleep at night knowing that someone might call you any second? Sounds worse than prison honestly.
    - daenney 10y ago
      > I'm surprised so many people agree to be on call. It's basically you letting a company hire you for 24 hours while being paid 9-5. Sounds like you need to renegotiate that. When I'm on call I don't get paid 9-5. There's a rather nice on-call bonus which pretty much means I get paid the full 24hrs. Might also depend on the country, some have laws around these kinds of things. For example, I can only be on-call for a 1 week stretch and have to be given a (paid) day off after. I can also not be on-call again for another couple of days. > I was never on call but I have a friend who was. Like you mentioned, it plays a huge role in how you live your life. That lingering feeling of they can call you any second haunts you day in and day out. If being on-call gives you this lingering feeling of doom then don't be on-call. If your systems are in that much of a fucked up state then you push back until that's sorted. Sometimes you need to let stuff burn. Being on-call does not have to equal being in a constant state of stress on the verge of panic attacks. > Even going to the movies must be a terrible experience? You either feel guilty for turning your pager off, or you spend the movie just ruminating about how it will end early for you if you get called. I don't go to the movies when I'm on-call. Turning your pager off at that point is irresponsible and not what I'm being paid for. In most companies you're only on-call for short periods at a time, it doesn't take over your life. You just go to the movies the day after, just like you'd plan any other activity and overlapping commitment. > How do you even sleep at night knowing that someone might call you any second? Sounds worse than prison honestly. By using Quiet Hours on your phone correctly and configuring the numbers that pages can come from to always go through. You get to bed and you sleep. The first few times you might sleep a bit less over it. Eventually you get the hang of it. A big part in this is knowing that you don't get paged for random crap but only ever when there's something truly wrong that can't wait until the next morning. Correctly behaving systems and not too trigger-happy alerting are crucial to this.
falcolas 10y ago
This can be done without having to call people in the middle of the night, by having distributed teams. When a company has people working regular shifts for all 24 hours, nobody is having to be woken up; someone is just working. I've lived this as a DBA, and it made so much more sense than expecting someone to be coherent and make the right decisions at 3am. The choice to put people to be on call is an explicitly sacrificing reliability for the sake of a budget. At what point is reliability worth more than the cost of a few remote employees?
- rincebrain 10y ago
  That's still an oncall rotation, just not outside of "work hours".
  - unethical_ban 10y ago
    When people complain about oncall, the off-hours portion is almost guaranteed to be the part complained about. Furthermore, I'd argue that if there is a dedicated group of people whose job it is SOLELY to work outages and help tickets, and their job is 100% that, it's not oncall. Oncall is "hey, you have these projects to do, but every 6 weeks you have to also answer every IM/phonecall/ticket escalation as well".
    - yebyen 10y ago
      Yes, I worked at a company where tech department was severely under-manned, and we had to be great at "SRE" type stuff. (I will definitely be reading this book, I even applied for this SRE position at Google while I was still working there...) The number one problem was that (starting in Year 4) there was nobody else with the knowledge to be on-call. So, I was on-call 24/7-365 for 5 years. Realistically it was a small enough company with localized (mostly east-coast USA) clients, that things going wrong in the middle of the night was pretty unusual. I had a great boss who mentored me to be able to handle these things by myself, and he mostly put the right tools in my hands so that it was possible to avoid getting woke in the middle of the night because anything went wrong short of a disk going bad. It happened often enough to be a good reason to want to leave! It was my first serious job, and for all its faults it was a pretty great job until my boss got a better offer. When he left, it seemed that there was no choice but to implement vSAN and VSphere. And then I actually didn't need to be woke by disks going bad in the middle of the night anymore. When I did go, there was nobody prepared to handle 75% of the incidents I would have been needed to work on. (I assume that many of those things just stay broken now when they go wrong.)

StreamBright 10y ago

You can be oncall from home. I found it perfect, just hanging around writing some code and get occasionally a page or two about somebody misunderstanding what production readiness means. :) On a more serious note, it is fun to work as an SRE, lots of problems you haven't seen before, many opportunities to learn about large scale systems. This sort of knowledge and view point that comes with it is invaluable for other companies too, you can move forward with your career faster after being an SRE for few years. (Amazon asks you do that for a year before you can move on to a different role).

szopa 10y ago
Yeah, being oncall for something like a well behaved backend system may be quite nice. I was an SRE for YouTube, and it was an almost constant bloodbath. YouTube's code changes at a relatively fast pace, there's a ton of developers, and it depends on a ton of different backends, and a problem with any of them would make us suffer. To make it worse I had a bit of bad luck, and was a magnet for weird and unlikely outages (bracing for my shift was a running joke in the team). So, if I was oncall at home it usually meant that shit started happening before breakfast and I didn't get enough quiet time to take my 10 minute bus ride to the office :) This said, I really enjoyed the experience. Yes, it was tiring and stressful, but it was also super interesting and exciting. Being responsible for such a huge site was incredible, and the feeling of figuring out how to overcome a big outage was exhilarating. I actually miss the pager drama from time to time (my wife does not).
- thewhitetulip 10y ago
  When you say outage does it mean youtube.com doesn't work at all, like reddit does some times or just few nodes of your load balancer give out?
sateesh 10y ago
I agree being an SRE exposes one to kind of problems that can occur in large scale production systems, the problems which one might not have thought during development or not unearthed during QA. But I think being oncall is flipside of being a SRE. To me being Oncall is oncall, be it be from home or oncall from office. If you are oncall during non-working hours, except what you save for the commute to being on office, your focus is drawn away from what you are doing (might be spending time with family, or book/movie you are enjoying, sleeping, or working on your project).