7 ms·

> Importantly, engineers should be on-call for their code - being on-call creates a positive feedback loop and makes it easier to know if their efforts in writi

> Importantly, engineers should be on-call for their code - being on-call creates a positive feedback loop and makes it easier to know if their efforts in writing production-ready code are paying off. I’ve heard people complain about the prospect of being on-call, so I’ll just ask this: if you’re not on-call for your code, who is? OK, that's fine, but in that case we can shut down the production system outside of business hours so that our work-life balance isn't affected. Oh? We can't shut down the production system outside of business hours? So we need to be on call continuously, 24/7, meaning we can't ever be off the grid or unavailable? That sounds like a we're expected to give up our personal lives at a moment's notice? Interesting, hmm.

AlphaSite 6y ago
Someone is always on call, so it’s just a case of whoa on the hook.
- Juliate 6y ago
  Yes, and in the long term, it's always better/more scalable/efficient to be disciplined enough to be able to have someone on call who has not designed/written the thing running.
  - edoceo 6y ago
    +1! And the feedback loop on this method is awesome too. Even in short term should have different parties for Dev and Ops
  - ownagefool 6y ago
    Why is that? The whole "you build it you run it" movement is an attempt to fix dev teams just not giving a fuck about quality of code they put out, especially from a reliability point of view. Why is the opposite better?
    - Juliate 6y ago
      It's a complement not an opposite. It's exactly like properly/cleverly documenting your code/project: not only for others now or in a few year, but also for yourself later on. It's having common rules across teams to get more reliability out of the whole company. You build it, you run it. Fine. Until the point when you can't anymore (because... reasons - it just happens). In any activity you want to sustain, you always have to have backups (in people and in processes), instead on relying on your-(self/team) alone.
    - pantulis 6y ago
      I guess because that enforces the documentation to be good enough so that someone without the faintest idea of what the software does can operate it. Probably this approach is more scalable specially in big companies where you can have operation teams on-call for a myriad of project. I personally believe that this does not guarantee a better service.
    - hyko 6y ago
      If your teams don’t give a fuck about the quality of their code, why would they give a fuck about the quality of their production support?
      - ownagefool 6y ago
        Because they'll be rang at 4am...
        Juliate 6y ago
        And what happens when they (or the most competent elements of them) leave? You will need to do in a rush what could have been prepared before. A whole business takes that into account as importantly as their disaster recovery processes (which is not necessarily something you focus early on, but you eventually do).
- user5994461 6y ago
  Factually wrong. All developers could turn off their phone/pagers and then there's no on call anymore! The worst that can happen is that the company is down a few hours overnight. Issues can be investigated and fixed during office hours. I'd wager that most companies don't have global customers and don't need 24/7 coverage.
  - sethammons 6y ago
    > the worst that can happen is the company is down for a few hours overnight I think this is a great example of why disagreements arise on HN: different world experiences and base assumptions. For many companies, being unavailable for that window of time would be catastrophic. We had one client that suffered about an hour of downtime (turned out to be their issue). They accounted that hour for 5 million dollars lost.
    - Too 6y ago
      $5M/hour can pay for a lot of engineers. So as you say - with such assumptions - you can and should pay for both up-front design, QA and people on-call, otherwise you only have your self to blame for the loss.
    - deleted 6y ago
      [deleted]
willejs 6y ago
If your writing software or operating infrastructure, you need to be on call. Otherwise you don’t have skin in the game. It makes you a better engineer, and at most places increases the quality of your software in two ways. One, you don’t want to get that call at 2am, so you think more about reliability, edge cases, writing playbooks etc. Secondly, when things do go wrong, you perform a post mortem and you get the action items in your stream of work. Additionally you should always track on call stats and use it as a metric in your team health checks. If people are getting called a lot out of hours, it’s time to pull the cord, and sort it out.
- Too 6y ago
  I'm generally for eating your own dogfood, but if we are to play devils advocate for a moment. If everyone is responsible for maintaining their own features, could it incentive you to not ship any features at all?
- yawaramin 6y ago
  But the decision to operate the service or infrastructure 24/7 was a business decision, it wasn't my decision. Why should I be on the hook for a decision I didn't make? And if the business really, really wants 24/7 availability, why should that cut into my personal time outside of work? Why shouldn't the business set up teams in multiple timezones for a follow-the-sun model? At the end of the day all this 'you need to be on call for your code' is purely a business money-saving ploy. We are an industry full of suckers, I guess, because we fell for the 'plausible-sounding' explanation hook, line, and sinker.
- tonyjstark 6y ago
  Is the project manager called too? Or is it just the developer who gets pressured in delivery and then pressured in maintenance, basically at the very low end of the food chain?
  - sethammons 6y ago
    We’ve done that: executive team, product owners, and managers. It was only for teams in a particular area of our product. System stability became a funded project.
arctangent 6y ago
> So we need to be on call continuously, 24/7, meaning we can't ever be off the grid or unavailable? The well-established solution to 24/7 availability is to operate a shift pattern.
- whateveracct 6y ago
  Yep and if you're on-call for a week 24/7, I'd say it's only fair you do nothing else preemptively. Because you need to allow for the possibility of being paged. No manager or employer would ever buy that shit because it rounds in the direction of less work though.
  - jodrellblank 6y ago
    A shift pattern without anyone on call. 3x 8hr shifts in a 24hr day[1], optionally distributed around the planet so that all shifts are working in their local daytime. [1] Gene Ray understood this.
OJFord 6y ago
Not 24/7/52, on a rota. And paid. It doesn't mean no work-life balance, it means part of the work is being available when on-call (limited 'life'), and life as normal when not. It's hardly some horrendous controversial idea, nor unique to software engineering.
- user5994461 6y ago
  >>> And paid The article did not mention anything about pay or compensation for oncall.
  - bdcravens 6y ago
    The article also didn't mention whether you are notified via email, SMS, or Slack; that seems like detail handled by the business.
    - yawaramin 6y ago
      When it comes to money, that is a rather important 'detail'. Especially given the fact that the most prevalent form of theft is wage theft: https://www.gq.com/story/wage-theft https://www.gq.com/story/wage-theft
      - bdcravens 6y ago
        For the purposes of the article, that's still a business concern. Presumably on-call expectations are part of the compensation agreement. (as they are with most industries)
        user5994461 6y ago
        I can't think of a single job/interview/offer where the expectations or the compensation for on call were discussed formally, let alone agreed. The best I get in job interviews is usually a mention that there is a rota every X period. Then have to poke interviewers trying to guess what is it like without coming up as too negative, "when is the last time you worked on a week end?" "when is the last time you were awaken in the middle of the night?"
gru 6y ago
No mature organisation would expect engineers to be on-call continuously, 24/7. There are ways to have a sane, balanced approach to on-call. See the SRE book for one example: https://landing.google.com/sre/sre-book/chapters/being-on-call/ https://landing.google.com/sre/sre-book/chapters/being-on-ca...
- yawaramin 6y ago
  So what's a reasonable on-call schedule for developers?
bdcravens 6y ago
Isn't that answered in your quote: "I’ve heard people complain about the prospect of being on-call, so I’ll just ask this: if you’re not on-call for your code, who is?" To me this implies an on-call rotation where you know your expectations. Not "we need to be on call continuously, 24/7, meaning we can't ever be off the grid or unavailable". Many other industries have the idea of being on-call, and they are "expected to give up our personal lives at a moment's notice" when they know they are on-call. (For example, my brother in law is a surgery tech; he's had to take off during family outings more than once) Also, if this happens often enough that it's a serious problem, this says a lot about the quality of the code you own.
- yawaramin 6y ago
  Code quality is a result of many factors, but the single biggest factor is whether Management treats developers as a profit centre or as a cost centre. That makes the difference in: - Developer compensation - Training and career development - Staffing properly i.e. not under-staffing - Giving devs proper slack time between tasks and not over-burdening them with projects - Letting developers own the stack not just in name only but truly own the technical decisions made in the stack without micro-management, including choice of language, platform, etc. Without all those factors, it's a red herring to point to the code quality. The code quality is just the final output of all of the above decisions.
lifeisstillgood 6y ago
There is always a cost to software in production. The issue is who pays and when? You can pay that cost upfront - for example JPL/NASA SDLC. This will ensure you won't get woken at odd hours but then the massive upfront cost is something most business won't pay You can sling code without tests and fix it in prod, hoping speed will help you find product market fit. Pretty much everyone sits somewhere between the two. This article just describes one point onthe spectrum where the author feels is best practise - but to be honest the trade offs vary across this spectrum. Probably the right way to think of this is "the total cost of making this software NASA level is 10X, and the revenue from such perfect working software would be 20X (with no loss due to downtime) As such of you ask me to not code to NASA standards, I and my team will incur a personal cost of 5X in being woken up, stressful release days etc. Therefore you will compensate me with payments of 5-10X. This discussion is much easier with a Union involved
- yawaramin 6y ago
  So developers who are on-call are paid 5-10x as much as JPL/NASA engineers?
  - lifeisstillgood 6y ago
    No. Ok - there is a spectrum of reliability - lets say that NASA produces the most reliable code anywhere, and that it has a very high cost to produce code like that. At the other end of the spectrum is some guy slinging php code out without any testing, hoping that it will turn into the next unicorn. If we asked both ends of the spectrum to write code to solve the same business (Pet food delivery app) then the guy slinging PHP will get woken up at 2am regularly because the server is always crashing. The NASA guy will never get woken up, but also the app probably will be out on the market a year after the first one. So the business has to choose a trade off - sling code and get lots of 2 am wake up calls or wait and possibly lose market share to a competitor. Now there was a famous example of a Reddit co-founder who slept next to his laptop and just rebooted the server every two hours till they discovered Python Supervisor. Now that seems ok - the business (co-founder) was making the trade off and exploiting the worker (same co-founder). The worker was happy to take the job because they were likely to get paid if it all worked out (and it did) The issue comes when the worker on call is not making the business judgement. How much should they demand in payment? If they have a healthy equity payment in a growth company, that might work just like the above founder. Otherwise the payment needs to come out of the money not spent. SO I guess my argument is that there is a fixed cost to reliable software to the business - it should either pay for highly reliable software, or it should pay the saved cash to the code clinger for each time the server goes down. This will change the trade off mathematics.
layoric 6y ago
I actually think it is a good feedback loop, but it needs to be staffed well and costs need to be well understood before a startup just takes on global 24/7 product and hoping the devs sort it out. I did 24/7 support solo and with just 1-2 other devs for years on global system and never again will I do DevOps in such a small team with on call 24/7 requirements. The cost of maintaining features and systems varies so that having great enterprise support can be a non issue or a constant headache that you have little to no control over (Eg a system that takes a dependency on external data you can’t control). On top of pushing features out constantly, maintaining quality and automating everything you can, a startup can easily fall into building systems their staff have trouble maintaining without impacting output significantly as well as impacting mental health of their devs. I think the problem is that it is hard to see these costs up front as you can build systems these days on cloud providers where most of the time things will come back on their own without intervention, but obviously depends what impact being offline for 5 mins vs 3 hours has on the business.

AlphaSite 6y ago

Someone is always on call, so it’s just a case of whoa on the hook.

Juliate 6y ago
Yes, and in the long term, it's always better/more scalable/efficient to be disciplined enough to be able to have someone on call who has not designed/written the thing running.
- edoceo 6y ago
  +1! And the feedback loop on this method is awesome too. Even in short term should have different parties for Dev and Ops
- ownagefool 6y ago
  Why is that? The whole "you build it you run it" movement is an attempt to fix dev teams just not giving a fuck about quality of code they put out, especially from a reliability point of view. Why is the opposite better?
  - Juliate 6y ago
    It's a complement not an opposite. It's exactly like properly/cleverly documenting your code/project: not only for others now or in a few year, but also for yourself later on. It's having common rules across teams to get more reliability out of the whole company. You build it, you run it. Fine. Until the point when you can't anymore (because... reasons - it just happens). In any activity you want to sustain, you always have to have backups (in people and in processes), instead on relying on your-(self/team) alone.
  - pantulis 6y ago
    I guess because that enforces the documentation to be good enough so that someone without the faintest idea of what the software does can operate it. Probably this approach is more scalable specially in big companies where you can have operation teams on-call for a myriad of project. I personally believe that this does not guarantee a better service.
  - hyko 6y ago
    If your teams don’t give a fuck about the quality of their code, why would they give a fuck about the quality of their production support?
    - ownagefool 6y ago
      Because they'll be rang at 4am...
      - Juliate 6y ago
        And what happens when they (or the most competent elements of them) leave? You will need to do in a rush what could have been prepared before. A whole business takes that into account as importantly as their disaster recovery processes (which is not necessarily something you focus early on, but you eventually do).
user5994461 6y ago
Factually wrong. All developers could turn off their phone/pagers and then there's no on call anymore! The worst that can happen is that the company is down a few hours overnight. Issues can be investigated and fixed during office hours. I'd wager that most companies don't have global customers and don't need 24/7 coverage.
- sethammons 6y ago
  > the worst that can happen is the company is down for a few hours overnight I think this is a great example of why disagreements arise on HN: different world experiences and base assumptions. For many companies, being unavailable for that window of time would be catastrophic. We had one client that suffered about an hour of downtime (turned out to be their issue). They accounted that hour for 5 million dollars lost.
  - Too 6y ago
    $5M/hour can pay for a lot of engineers. So as you say - with such assumptions - you can and should pay for both up-front design, QA and people on-call, otherwise you only have your self to blame for the loss.
  - deleted 6y ago
    [deleted]

willejs 6y ago

If your writing software or operating infrastructure, you need to be on call. Otherwise you don’t have skin in the game. It makes you a better engineer, and at most places increases the quality of your software in two ways. One, you don’t want to get that call at 2am, so you think more about reliability, edge cases, writing playbooks etc. Secondly, when things do go wrong, you perform a post mortem and you get the action items in your stream of work. Additionally you should always track on call stats and use it as a metric in your team health checks. If people are getting called a lot out of hours, it’s time to pull the cord, and sort it out.

Too 6y ago
I'm generally for eating your own dogfood, but if we are to play devils advocate for a moment. If everyone is responsible for maintaining their own features, could it incentive you to not ship any features at all?
yawaramin 6y ago
But the decision to operate the service or infrastructure 24/7 was a business decision, it wasn't my decision. Why should I be on the hook for a decision I didn't make? And if the business really, really wants 24/7 availability, why should that cut into my personal time outside of work? Why shouldn't the business set up teams in multiple timezones for a follow-the-sun model? At the end of the day all this 'you need to be on call for your code' is purely a business money-saving ploy. We are an industry full of suckers, I guess, because we fell for the 'plausible-sounding' explanation hook, line, and sinker.
tonyjstark 6y ago
Is the project manager called too? Or is it just the developer who gets pressured in delivery and then pressured in maintenance, basically at the very low end of the food chain?
- sethammons 6y ago
  We’ve done that: executive team, product owners, and managers. It was only for teams in a particular area of our product. System stability became a funded project.

arctangent 6y ago

> So we need to be on call continuously, 24/7, meaning we can't ever be off the grid or unavailable? The well-established solution to 24/7 availability is to operate a shift pattern.

whateveracct 6y ago
Yep and if you're on-call for a week 24/7, I'd say it's only fair you do nothing else preemptively. Because you need to allow for the possibility of being paged. No manager or employer would ever buy that shit because it rounds in the direction of less work though.
- jodrellblank 6y ago
  A shift pattern without anyone on call. 3x 8hr shifts in a 24hr day[1], optionally distributed around the planet so that all shifts are working in their local daytime. [1] Gene Ray understood this.

OJFord 6y ago

Not 24/7/52, on a rota. And paid. It doesn't mean no work-life balance, it means part of the work is being available when on-call (limited 'life'), and life as normal when not. It's hardly some horrendous controversial idea, nor unique to software engineering.

user5994461 6y ago
>>> And paid The article did not mention anything about pay or compensation for oncall.
- bdcravens 6y ago
  The article also didn't mention whether you are notified via email, SMS, or Slack; that seems like detail handled by the business.
  - yawaramin 6y ago
    When it comes to money, that is a rather important 'detail'. Especially given the fact that the most prevalent form of theft is wage theft: https://www.gq.com/story/wage-theft https://www.gq.com/story/wage-theft
    - bdcravens 6y ago
      For the purposes of the article, that's still a business concern. Presumably on-call expectations are part of the compensation agreement. (as they are with most industries)
      - user5994461 6y ago
        I can't think of a single job/interview/offer where the expectations or the compensation for on call were discussed formally, let alone agreed. The best I get in job interviews is usually a mention that there is a rota every X period. Then have to poke interviewers trying to guess what is it like without coming up as too negative, "when is the last time you worked on a week end?" "when is the last time you were awaken in the middle of the night?"

gru 6y ago

No mature organisation would expect engineers to be on-call continuously, 24/7. There are ways to have a sane, balanced approach to on-call. See the SRE book for one example: https://landing.google.com/sre/sre-book/chapters/being-on-call/ https://landing.google.com/sre/sre-book/chapters/being-on-ca...

yawaramin 6y ago
So what's a reasonable on-call schedule for developers?

bdcravens 6y ago

Isn't that answered in your quote: "I’ve heard people complain about the prospect of being on-call, so I’ll just ask this: if you’re not on-call for your code, who is?" To me this implies an on-call rotation where you know your expectations. Not "we need to be on call continuously, 24/7, meaning we can't ever be off the grid or unavailable". Many other industries have the idea of being on-call, and they are "expected to give up our personal lives at a moment's notice" when they know they are on-call. (For example, my brother in law is a surgery tech; he's had to take off during family outings more than once) Also, if this happens often enough that it's a serious problem, this says a lot about the quality of the code you own.

yawaramin 6y ago
Code quality is a result of many factors, but the single biggest factor is whether Management treats developers as a profit centre or as a cost centre. That makes the difference in: - Developer compensation - Training and career development - Staffing properly i.e. not under-staffing - Giving devs proper slack time between tasks and not over-burdening them with projects - Letting developers own the stack not just in name only but truly own the technical decisions made in the stack without micro-management, including choice of language, platform, etc. Without all those factors, it's a red herring to point to the code quality. The code quality is just the final output of all of the above decisions.

lifeisstillgood 6y ago

There is always a cost to software in production. The issue is who pays and when? You can pay that cost upfront - for example JPL/NASA SDLC. This will ensure you won't get woken at odd hours but then the massive upfront cost is something most business won't pay You can sling code without tests and fix it in prod, hoping speed will help you find product market fit. Pretty much everyone sits somewhere between the two. This article just describes one point onthe spectrum where the author feels is best practise - but to be honest the trade offs vary across this spectrum. Probably the right way to think of this is "the total cost of making this software NASA level is 10X, and the revenue from such perfect working software would be 20X (with no loss due to downtime) As such of you ask me to not code to NASA standards, I and my team will incur a personal cost of 5X in being woken up, stressful release days etc. Therefore you will compensate me with payments of 5-10X. This discussion is much easier with a Union involved

yawaramin 6y ago
So developers who are on-call are paid 5-10x as much as JPL/NASA engineers?
- lifeisstillgood 6y ago
  No. Ok - there is a spectrum of reliability - lets say that NASA produces the most reliable code anywhere, and that it has a very high cost to produce code like that. At the other end of the spectrum is some guy slinging php code out without any testing, hoping that it will turn into the next unicorn. If we asked both ends of the spectrum to write code to solve the same business (Pet food delivery app) then the guy slinging PHP will get woken up at 2am regularly because the server is always crashing. The NASA guy will never get woken up, but also the app probably will be out on the market a year after the first one. So the business has to choose a trade off - sling code and get lots of 2 am wake up calls or wait and possibly lose market share to a competitor. Now there was a famous example of a Reddit co-founder who slept next to his laptop and just rebooted the server every two hours till they discovered Python Supervisor. Now that seems ok - the business (co-founder) was making the trade off and exploiting the worker (same co-founder). The worker was happy to take the job because they were likely to get paid if it all worked out (and it did) The issue comes when the worker on call is not making the business judgement. How much should they demand in payment? If they have a healthy equity payment in a growth company, that might work just like the above founder. Otherwise the payment needs to come out of the money not spent. SO I guess my argument is that there is a fixed cost to reliable software to the business - it should either pay for highly reliable software, or it should pay the saved cash to the code clinger for each time the server goes down. This will change the trade off mathematics.

layoric 6y ago

I actually think it is a good feedback loop, but it needs to be staffed well and costs need to be well understood before a startup just takes on global 24/7 product and hoping the devs sort it out. I did 24/7 support solo and with just 1-2 other devs for years on global system and never again will I do DevOps in such a small team with on call 24/7 requirements. The cost of maintaining features and systems varies so that having great enterprise support can be a non issue or a constant headache that you have little to no control over (Eg a system that takes a dependency on external data you can’t control). On top of pushing features out constantly, maintaining quality and automating everything you can, a startup can easily fall into building systems their staff have trouble maintaining without impacting output significantly as well as impacting mental health of their devs. I think the problem is that it is hard to see these costs up front as you can build systems these days on cloud providers where most of the time things will come back on their own without intervention, but obviously depends what impact being offline for 5 mins vs 3 hours has on the business.