5 ms·

Ask HN: Options for back end engineer sick of oncall

13 points by lknfrnpt 4y ago

Hi folks, I'm close to hitting a decade of full time experience. In that time, I have always worked in back end engineering roles requiring oncall duties. I've recently identified a few things that are my key stressors at work, and the key contributors to my burn out. The top of that list is the existence of oncall duties, and the need to be ready to respond for a week at a time. Even with a quiet oncall, I still plan my personal life around being oncall. Same goes for planning any tie off. I've reached a point where I don't need or want to self sacrifice (e.g,. for total comp, or prestige, or whatever); instead, I'm trying to tweak my path forward for the next decade or so of work. I'm not ruling out a change of disciplines, for instance mobile or frontend development. That said, I have a strong set of skills and deep experience from the last decade. So I'm wondering, what sorts of roles might be less of a skill reset than what I generalize as "back end" engineering? Some background on me, if it helps: * Management is doable but not enjoyable (I dislike the context switching, miss building things, and can stomach process as an IC but don't enjoy driving process). * Years of experience in security engineering * Some enjoyment of using basic data science (ETLs, Jupyter notebooks analyzing warehoused data, etc) to inform eng decisions in the past * Some low level Linux programming experience for about a year or two, but pretty surface level ("understands procfs") * Of course, the rest is all just building CRUD apps hitting a variety of database types and generally doing so in products with millions of customers, where downtime or breaking changes are not tolerated. Pretty vanilla stuff. Distributed systems to the extent of "distributed CRUD" (lol) When times are good, I'm very happy being an IC engineer. I can handle the demands of engineering work and the organizational challenges that come with it, when contained to a reasonable business day. I'm just worried that I can't sustain what I've been doing for so long (exacerbated by being oncall quite regularly in my current role) and don't want to sacrifice my personal well being (or family's) any more. I'd rather figure out a new role than do something drastic like quitting out of burn out, only to start again in another backend role that basically has the same problems as my current situation. Thanks for reading!

aregsar 4y ago
Try to transition from developer role to a system architect role.
- lknfrnpt 4y ago
  At the companies I've worked, there are no architect roles; architecture is the responsibility of ICs pretty early on (in addition to implementation and ownership of operation). I've seen this at both small startups and large FANG-like companies. Any companies in mind with this sort of role? Thanks!
gregjor 4y ago
Look at what causes the support calls and fix the problems. Ideally the apps you work on shouldn't have downtime or breaking changes. I know things don't always go to plan, but you can address getting frequent calls to deal with preventable problems. I currently support three web apps for different customers, and I will get called if something blows up. I have monitoring on the apps and cloud resources so I usually find out about problems before my customers or their users do. I have implemented deployment processes to prevent breaking changes as much as possible, and rollback/recovery processes just in case. I get two or three emergency calls a year.
- tkiolp4 4y ago
  The problem of being on-call is not only the potential issues that can appear at 3am in the morning. For me, at least, the worst part is the fact that I cannot do anything outside home like going out for a run, or to to the cinema, or for a walk with my friends. I cannot even go for a 30 min walk without taking my laptop!
  - gregjor 4y ago
    I get calls, messages, emails on my phone. I don't tell customers I will sit in front of my screen 24/7. I tell them one hour or less. If they need faster response than that they will have to plan for it and pay for it. I'm guessing you have a f/t job with on-call required. Do you really need to have your laptop on-hand every minute? I have had jobs in the past with managers who told me I would have to do on-call duty. I told them no, my kids and personal life mean more to me. I agreed to carry a phone or pager, and get to problems as fast as I could, but not let on-call intrude so much I was effectively working extra hours for free. I never got penalized or fired for setting some limits. I never knew a manager or executive who was on call 24/7 or carried their laptop around at the golf course.
    - tkiolp4 4y ago
      I’m not on call, but when doing interviews I explicitly ask about the on-call setup (if any). The usual setup (for backend engineers) is: - 1 week in-call every 5 to 6 weeks (depending on how many engineers are available for on call) - if you get paged, you need to acknowledge within 10-15 min. Sometimes it’s nothing and you can go back to sleep, but sometimes you may need your laptop to fix things. The point is: you don’t know before hand - it’s paid. In my opinion the money is not worth it - it’s mandatory. But even if it’s not, if all your teammates are on call and you’re the only one who refuses it, well, that usually creates friction
stevenalowe 4y ago
Sympathies. The way on-call is handled at most companies is a health nightmare (regular shift plus on call).
detaro 4y ago
Serious question, is on-call really that ubiquitous for devs in companies running services that not wanting to do it means "change to different role"?
- lknfrnpt 4y ago
  If you work on backend at a small to medium sized organizations, it is pretty common to be expected to own what you ship beyond landing pull requests: deploying, operating, migrating, oncall, etc. (Yes)
- Viliam1234 4y ago
  It seems to me like a consequence of the ongoing unification of all software development related jobs. In the ancient times, when you had separate roles for programmers, testers, system administrators, etc., all working in the waterfall environment, there was no need for a programmer to be on call. The programmers developed during their working hours. When the product was ready, it was tested by the testers. When it finally got to production, it was mostly bug-free. You had the system administrators on call, in case some server caught fire. After the testers were mostly eliminated, bugs in production became more frequent. Continuous deployment into production requires continuous on-call duty. Devops means that the programmers are now also responsible for the failures of servers. Now the programmers are expected to also be agile in their free time. (But of course, we are collectively all too smart and too proud to form a union. I guess we have to suffer the obvious consequences then.)
comprev 4y ago
Stating the obvious, perhaps find a role which requires zero on-call? None of my backend (Java/Python) colleagues do any on-call, and sticking to contracted hours (09:00-17:30) is encouraged by team leads. I've done a fair amount of on-call over the years and now pivoted from Ops handling production to internal projects. System uptime simply isn't my responsibility any more :)
- lknfrnpt 4y ago
  I'd love to dig into this a little bit. Any ideas? The hypothetical examples that stand out to me are security engineering type roles in which the teams in question might not actually own the operations of infrastructure they build/maintain. I know a tiny set of people who work on operating systems at various levels for consumer devices (i.e., Android these days). How prevalent are those jobs? How do they stack up in terms of comp and hiring demand?
aristofun 4y ago
Most probably you deeply don’t care about the company’s mission in general. You just enjoy some specific activities, being a professional. The more fellow programmers i see, the more im convinced it’s not enough for such a demanding job. You have to personally care to some degree about your users, about the general problem you’re solving, you need to feel meaning in what you do. Then oncalls etc also start show some meaning and value to the bigger picture. Without it — you ll face plenty of energy sinks in any discipline.
- tkiolp4 4y ago
  Your theory doesn’t work… otherwise why managers are not on-call? You’ll say “because they cannot fix shit if needed”, but the real reason is: on-call sucks, you cannot force managers to do it, but you can force engineers. I’m in the same situation as OP, and it seems they and I, we are just normal professionals who care about their career, but are not willing to sacrifice our scarce free time in exchange for more money or whatever.
  - mousetree 4y ago
    Why do you think the real reason is not “because they cannot fix shit if needed”. Seems the most reasonable to me.
    - tkiolp4 4y ago
      At least in my experience, managers (engineering managers, sorry for not being more explicit. I didn’t mean product managers or similar) have a good number of years of experience as software engineers before turning into managers. They are not clueless about the products we built and maintain. So, even if junior engineers are on-call, why on earth an engineer manager with over 10 years of coding experience cannot be on call as well? In my experience, the answer is: because being oncall is shit, and upper management cannot force middle level managers to be on call.
      - tra3 4y ago
toomuchtodo 4y ago
Go into infosec with a tilt towards app sec and architecture. No on call (unless you’re part of a SOC or incident response is part of the role). Do not accept on call roles. I left DevOps/infra for infosec/risk mgmt in part due to on call rotations, and would never accept a job again that required them.
- tkiolp4 4y ago
  Isn’t infosec more prone to emergencies?E.g., we have been hacked! Please help, or we are being DDOSed, etc. If any given company has an infosec team is because they are big enough that they are targets. Also, fixing security issues sounds to me worse than “just” fixing code issues.
  - lknfrnpt 4y ago
    In my experience infosec is more preventative; writing secure code reused by a larger org, reviewing designs from non security teams, maintaining automated monitoring and yes when it happens responding to incidents. Rarely just fixing code issues! Doesn't sound so bad as I write it and remember it... :)
DarrenDev 4y ago
I think you just need to find a company where devs do not do oncall. Companies that have a dedicated "production support" team or something similarly named that handle maintaining production environments. I've worked for a dozen different companies (contractor) over the past 10 years as a backend engineer and have never been faced with oncall. Neither have my permanent employee colleagues. In many cases we had no direct access to production environments as that was managed by separate teams. In the one startup I worked at, it was so early stage that there were only ever a half dozen users, and the CTO handled any issues that popped up over the weekend. The type of companies that had dedicated teams for maintaining live environments included big banks, government departments, insurance companies, large healthcare companies.
- Olphs 4y ago
  >dedicated "production support" team That's interesting, so if some bug pops up during the weekend they will just revert back? And then that bug gets handled later. Or is there also so rigorous QA process that bugs rarely happen, and it's more like infra issues that the "product support" team can handle on their own? The interesting part is that I think on-call is not only about infra issues etc., but could also be bugs in the application or some other weird behavior, which I'm assuming the "production support" wouldn't know how to handle. Maybe there is some huge amount of process to handle all that I'm just oblivious of, never having heard of this before?

13 points by lknfrnpt 4y ago

aregsar 4y ago

Try to transition from developer role to a system architect role.

lknfrnpt 4y ago
At the companies I've worked, there are no architect roles; architecture is the responsibility of ICs pretty early on (in addition to implementation and ownership of operation). I've seen this at both small startups and large FANG-like companies. Any companies in mind with this sort of role? Thanks!

gregjor 4y ago

Look at what causes the support calls and fix the problems. Ideally the apps you work on shouldn't have downtime or breaking changes. I know things don't always go to plan, but you can address getting frequent calls to deal with preventable problems. I currently support three web apps for different customers, and I will get called if something blows up. I have monitoring on the apps and cloud resources so I usually find out about problems before my customers or their users do. I have implemented deployment processes to prevent breaking changes as much as possible, and rollback/recovery processes just in case. I get two or three emergency calls a year.

tkiolp4 4y ago
The problem of being on-call is not only the potential issues that can appear at 3am in the morning. For me, at least, the worst part is the fact that I cannot do anything outside home like going out for a run, or to to the cinema, or for a walk with my friends. I cannot even go for a 30 min walk without taking my laptop!
- gregjor 4y ago
  I get calls, messages, emails on my phone. I don't tell customers I will sit in front of my screen 24/7. I tell them one hour or less. If they need faster response than that they will have to plan for it and pay for it. I'm guessing you have a f/t job with on-call required. Do you really need to have your laptop on-hand every minute? I have had jobs in the past with managers who told me I would have to do on-call duty. I told them no, my kids and personal life mean more to me. I agreed to carry a phone or pager, and get to problems as fast as I could, but not let on-call intrude so much I was effectively working extra hours for free. I never got penalized or fired for setting some limits. I never knew a manager or executive who was on call 24/7 or carried their laptop around at the golf course.
  - tkiolp4 4y ago
    I’m not on call, but when doing interviews I explicitly ask about the on-call setup (if any). The usual setup (for backend engineers) is: - 1 week in-call every 5 to 6 weeks (depending on how many engineers are available for on call) - if you get paged, you need to acknowledge within 10-15 min. Sometimes it’s nothing and you can go back to sleep, but sometimes you may need your laptop to fix things. The point is: you don’t know before hand - it’s paid. In my opinion the money is not worth it - it’s mandatory. But even if it’s not, if all your teammates are on call and you’re the only one who refuses it, well, that usually creates friction

stevenalowe 4y ago

Sympathies. The way on-call is handled at most companies is a health nightmare (regular shift plus on call).

detaro 4y ago

Serious question, is on-call really that ubiquitous for devs in companies running services that not wanting to do it means "change to different role"?

lknfrnpt 4y ago
If you work on backend at a small to medium sized organizations, it is pretty common to be expected to own what you ship beyond landing pull requests: deploying, operating, migrating, oncall, etc. (Yes)
Viliam1234 4y ago
It seems to me like a consequence of the ongoing unification of all software development related jobs. In the ancient times, when you had separate roles for programmers, testers, system administrators, etc., all working in the waterfall environment, there was no need for a programmer to be on call. The programmers developed during their working hours. When the product was ready, it was tested by the testers. When it finally got to production, it was mostly bug-free. You had the system administrators on call, in case some server caught fire. After the testers were mostly eliminated, bugs in production became more frequent. Continuous deployment into production requires continuous on-call duty. Devops means that the programmers are now also responsible for the failures of servers. Now the programmers are expected to also be agile in their free time. (But of course, we are collectively all too smart and too proud to form a union. I guess we have to suffer the obvious consequences then.)

comprev 4y ago

Stating the obvious, perhaps find a role which requires zero on-call? None of my backend (Java/Python) colleagues do any on-call, and sticking to contracted hours (09:00-17:30) is encouraged by team leads. I've done a fair amount of on-call over the years and now pivoted from Ops handling production to internal projects. System uptime simply isn't my responsibility any more :)

lknfrnpt 4y ago
I'd love to dig into this a little bit. Any ideas? The hypothetical examples that stand out to me are security engineering type roles in which the teams in question might not actually own the operations of infrastructure they build/maintain. I know a tiny set of people who work on operating systems at various levels for consumer devices (i.e., Android these days). How prevalent are those jobs? How do they stack up in terms of comp and hiring demand?

aristofun 4y ago

Most probably you deeply don’t care about the company’s mission in general. You just enjoy some specific activities, being a professional. The more fellow programmers i see, the more im convinced it’s not enough for such a demanding job. You have to personally care to some degree about your users, about the general problem you’re solving, you need to feel meaning in what you do. Then oncalls etc also start show some meaning and value to the bigger picture. Without it — you ll face plenty of energy sinks in any discipline.

tkiolp4 4y ago
Your theory doesn’t work… otherwise why managers are not on-call? You’ll say “because they cannot fix shit if needed”, but the real reason is: on-call sucks, you cannot force managers to do it, but you can force engineers. I’m in the same situation as OP, and it seems they and I, we are just normal professionals who care about their career, but are not willing to sacrifice our scarce free time in exchange for more money or whatever.
- mousetree 4y ago
  Why do you think the real reason is not “because they cannot fix shit if needed”. Seems the most reasonable to me.
  - tkiolp4 4y ago
    At least in my experience, managers (engineering managers, sorry for not being more explicit. I didn’t mean product managers or similar) have a good number of years of experience as software engineers before turning into managers. They are not clueless about the products we built and maintain. So, even if junior engineers are on-call, why on earth an engineer manager with over 10 years of coding experience cannot be on call as well? In my experience, the answer is: because being oncall is shit, and upper management cannot force middle level managers to be on call.
    - tra3 4y ago

toomuchtodo 4y ago

Go into infosec with a tilt towards app sec and architecture. No on call (unless you’re part of a SOC or incident response is part of the role). Do not accept on call roles. I left DevOps/infra for infosec/risk mgmt in part due to on call rotations, and would never accept a job again that required them.

tkiolp4 4y ago
Isn’t infosec more prone to emergencies?E.g., we have been hacked! Please help, or we are being DDOSed, etc. If any given company has an infosec team is because they are big enough that they are targets. Also, fixing security issues sounds to me worse than “just” fixing code issues.
- lknfrnpt 4y ago
  In my experience infosec is more preventative; writing secure code reused by a larger org, reviewing designs from non security teams, maintaining automated monitoring and yes when it happens responding to incidents. Rarely just fixing code issues! Doesn't sound so bad as I write it and remember it... :)

DarrenDev 4y ago

I think you just need to find a company where devs do not do oncall. Companies that have a dedicated "production support" team or something similarly named that handle maintaining production environments. I've worked for a dozen different companies (contractor) over the past 10 years as a backend engineer and have never been faced with oncall. Neither have my permanent employee colleagues. In many cases we had no direct access to production environments as that was managed by separate teams. In the one startup I worked at, it was so early stage that there were only ever a half dozen users, and the CTO handled any issues that popped up over the weekend. The type of companies that had dedicated teams for maintaining live environments included big banks, government departments, insurance companies, large healthcare companies.

Olphs 4y ago
>dedicated "production support" team That's interesting, so if some bug pops up during the weekend they will just revert back? And then that bug gets handled later. Or is there also so rigorous QA process that bugs rarely happen, and it's more like infra issues that the "product support" team can handle on their own? The interesting part is that I think on-call is not only about infra issues etc., but could also be bugs in the application or some other weird behavior, which I'm assuming the "production support" wouldn't know how to handle. Maybe there is some huge amount of process to handle all that I'm just oblivious of, never having heard of this before?