6 ms·

The industry myth of "devs need to be on-call just in case prod crashes at 3am" needs to die. First, a system failure addressed by someone awoken "at 3am" assu

The industry myth of "devs need to be on-call just in case prod crashes at 3am" needs to die. First, a system failure addressed by someone awoken "at 3am" assumes said person can transition from sleep to peek analytic mode in moments. This is obviously not the case. Second, a system failure addressed by someone awoken "at 3am" assumes said person possesses omniscient awareness of all aspects of a non-trivial system. This is obviously not the case. Third, and finally, a system failure addressed by someone "at 3am" assumes said person can rectify the problem in real time and without any coworker collaboration nor stakeholder input. The last is most important: If a resolution requires a system change, what organization would allow one person at 3am to make it and push to prod?

hn_user82179 2y ago
great points. I think over the 8 years of my SRE experience, I've probably caused a few outages after being paged at 3-4am and/or prolonged them. I've fallen asleep during one responding time (fortunately the main outage had been fixed but we went way beyond our internal SLO for backups as I had fallen asleep before running them). That said, the author neglected to mention timezones or following the sun. A 12/12 hour shift or 8/8/8 (NA/EU, NA/EU/EMEA) addresses the sleep deprivation problem pretty well, and is pretty easy to staff in a large enough org
- AdieuToLogic 2y ago
  > That said, the author neglected to mention timezones or following the sun. A 12/12 hour shift or 8/8/8 (NA/EU, NA/EU/EMEA) addresses the sleep deprivation problem pretty well I completely agree. The "get a call at 3am" scenario, to me, is shorthand for an organization which intentionally under-staffs in order to save money. If a system has genuine 24x7 support needs, be it SLA's or inferior construction, it is incumbent upon the organization to staff accordingly. Still and all, identifying a production issue is one thing, expecting near real-time resolution by personnel likely unfamiliar with the intricacies of every aspect of a non-trivial production system is another.
  - nijave 2y ago
    It always amazes me these places that have "24/7 support needs" then all of a sudden a bug comes in that's "not important" even though it has customer uptime impacts.
whstl 2y ago
Great points. > If a resolution requires a system change, what organization would allow one person at 3am to make it and push to prod? The only advantage of having a micromanaging manager: if they want to review all PRs and system changes, they also must be on call every single night of the year.
procaryote 2y ago
If the code is important enough that it needs to be fixed if it breaks at 03:00, someone needs to be on-call If that someone isn't the dev, things breaking at 03:00 is someone else's problem from the PoV of the dev, and it will likely keep breaking. I've tried at several companies to get dev-teams to prioritise things causing a lot of work for the ops-team, and nothing has worked as well as disbanding the ops team and putting the devs on-call. Pain from a problem needs to live where the problem can be fixed.
- inetknght 2y ago
  > If that someone isn't the dev, things breaking at 03:00 is someone else's problem from the PoV of the dev, and it will likely keep breaking. If that's the case, then the person responding at 03:00 needs to have a serious conversation with the dev, or with whoever's assigning/prioritizing tickets to the dev.
  - SpicyLemonZest 2y ago
    It's not usually tickets that are needed, but a complex set of software development and reliability practices that can't be communicated through Jira.
- AdieuToLogic 2y ago
  > I've tried at several companies to get dev-teams to prioritise things causing a lot of work for the ops-team, and nothing has worked as well as disbanding the ops team and putting the devs on-call. This is a leadership problem, not a development problem. If stakeholders prioritize resolving production issues and development teams do not address the objectives set forth, then there are bigger problems afoot. > Pain from a problem needs to live where the problem can be fixed. Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0]. 0 - https://nielsbohrmann.com/prices-law/ https://nielsbohrmann.com/prices-law/
  - procaryote 2y ago
    It's only a leadership problem if you split the responsibility for the thing so you have A: a team that builds and maintains it and B: someone else who has to wake up at night and fix it. If A and B are the same people, very little leadership is needed as they are innately motivated, and have direct experience. > Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0]. It's not punitive management. Someone will have to work when the thing breaks. It can be the people who can fix the problem, or it can be someone else.
markus_zhang 2y ago
Well during on-call I think usually there is a chain of wake-ups. The on-call guy tries to fix the issue but finds out he is not familiar with it so creates a channel and wakes up the next guy in the right team, and so on. The first guy usually just wakes up to "acknowledge" that someone is taking care of the issue, and writes the wrap-ups and closes the channel.
rob74 2y ago
> If a resolution requires a system change, what organization would allow one person at 3am to make it and push to prod? Well yeah, if you have on call, I guess you have to trust the on-call person to make that decision. If they get called at 3am, the problem is obviously so bad that almost anything they do can only make it better, right? If it's some overlooked edge case that crashes the whole application and can be fixed with a one line change, it's probably Ok to do it a 3am. If it's more complicated, maybe wait until 7am and discuss it with others too. And, in any case, a thorough post mortem (during office hours) and checking if the 3am fix is really a proper fix for the issue is in order...
jjice 2y ago
The way I was taught to be on call by a guy I worked with that was also an SRE at a large software company was to "patch the hole in the tire and get it to the service center". It wasn't about fixing the problem or fully understanding it, but instead making sure the system can run for the time being, get some sleep, and have a more complete triage in the morning. I've found this to work pretty well the past few years. While I do agree with your sentiment, I'd say that the perspective I've learned about being on call is a big different than the one you've experienced (which may be the more common one, I'm not sure). So this may end up being a cultural difference between on-call situations.
- AdieuToLogic 2y ago
  > The way I was taught to be on call by a guy I worked with that was also an SRE at a large software company was to "patch the hole in the tire and get it to the service center". That is a great philosophy to have when supporting a prod system off-hours, one which I fully subscribe. > While I do agree with your sentiment, I'd say that the perspective I've learned about being on call is a big[sic] different than the one you've experienced (which may be the more common one, I'm not sure). My underlying thesis is not with being on-call, but instead expecting a developer to perform their non-support duties in addition to being on-call. The worst case scenario of this is when the on-call developer is also tier one support. If an organization wants to have developers perform SRE duties, presumably due to deeper understanding of a system, fine. Assign them to support and suspend development responsibilities during same. Just my $0.02
- asdf6969 2y ago
  In my experience we patch the hole in the tire and then get directed to patch a hole in another tire. Manager says work on the highest priority tire so now there are 2 partially broken tires but the problem is technically fixed. Repeat for 10 years and now it’s time to rebuild it “correctly”, but actually we just have two cars to patch since nobody knows what awful hacks we need to port over for “expected” behaviour
jyounker 2y ago
Devs need to be on call at three AM so that they suffer when their software fails. This is how you align developer motivation with operational motivation. The organization needs to incur financial penalties when the on-call staff have to respond at three AM. This aligns the organizations motivations with the operational motivations. When managers see an on-call-incident line-item going up then they're more willing to tell product to take a hike.
- AdieuToLogic 2y ago
  > Devs need to be on call at three AM so that they suffer when their software fails. This is how you align developer motivation with operational motivation. As I mentioned in a peer response: Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0]. 0 - https://nielsbohrmann.com/prices-law/ https://nielsbohrmann.com/prices-law/
  - jyounker 2y ago
    It's interesting that you see this as being a punitive policy. Our job is producing quality systems. If our software is so bug ridden that caring for it is a punishment, then failed we've at producing quality software.

hn_user82179 2y ago

great points. I think over the 8 years of my SRE experience, I've probably caused a few outages after being paged at 3-4am and/or prolonged them. I've fallen asleep during one responding time (fortunately the main outage had been fixed but we went way beyond our internal SLO for backups as I had fallen asleep before running them). That said, the author neglected to mention timezones or following the sun. A 12/12 hour shift or 8/8/8 (NA/EU, NA/EU/EMEA) addresses the sleep deprivation problem pretty well, and is pretty easy to staff in a large enough org

AdieuToLogic 2y ago
> That said, the author neglected to mention timezones or following the sun. A 12/12 hour shift or 8/8/8 (NA/EU, NA/EU/EMEA) addresses the sleep deprivation problem pretty well I completely agree. The "get a call at 3am" scenario, to me, is shorthand for an organization which intentionally under-staffs in order to save money. If a system has genuine 24x7 support needs, be it SLA's or inferior construction, it is incumbent upon the organization to staff accordingly. Still and all, identifying a production issue is one thing, expecting near real-time resolution by personnel likely unfamiliar with the intricacies of every aspect of a non-trivial production system is another.
- nijave 2y ago
  It always amazes me these places that have "24/7 support needs" then all of a sudden a bug comes in that's "not important" even though it has customer uptime impacts.

whstl 2y ago

Great points. > If a resolution requires a system change, what organization would allow one person at 3am to make it and push to prod? The only advantage of having a micromanaging manager: if they want to review all PRs and system changes, they also must be on call every single night of the year.

procaryote 2y ago

If the code is important enough that it needs to be fixed if it breaks at 03:00, someone needs to be on-call If that someone isn't the dev, things breaking at 03:00 is someone else's problem from the PoV of the dev, and it will likely keep breaking. I've tried at several companies to get dev-teams to prioritise things causing a lot of work for the ops-team, and nothing has worked as well as disbanding the ops team and putting the devs on-call. Pain from a problem needs to live where the problem can be fixed.

inetknght 2y ago
> If that someone isn't the dev, things breaking at 03:00 is someone else's problem from the PoV of the dev, and it will likely keep breaking. If that's the case, then the person responding at 03:00 needs to have a serious conversation with the dev, or with whoever's assigning/prioritizing tickets to the dev.
- SpicyLemonZest 2y ago
  It's not usually tickets that are needed, but a complex set of software development and reliability practices that can't be communicated through Jira.
AdieuToLogic 2y ago
> I've tried at several companies to get dev-teams to prioritise things causing a lot of work for the ops-team, and nothing has worked as well as disbanding the ops team and putting the devs on-call. This is a leadership problem, not a development problem. If stakeholders prioritize resolving production issues and development teams do not address the objectives set forth, then there are bigger problems afoot. > Pain from a problem needs to live where the problem can be fixed. Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0]. 0 - https://nielsbohrmann.com/prices-law/ https://nielsbohrmann.com/prices-law/
- procaryote 2y ago
  It's only a leadership problem if you split the responsibility for the thing so you have A: a team that builds and maintains it and B: someone else who has to wake up at night and fix it. If A and B are the same people, very little leadership is needed as they are innately motivated, and have direct experience. > Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0]. It's not punitive management. Someone will have to work when the thing breaks. It can be the people who can fix the problem, or it can be someone else.

markus_zhang 2y ago

Well during on-call I think usually there is a chain of wake-ups. The on-call guy tries to fix the issue but finds out he is not familiar with it so creates a channel and wakes up the next guy in the right team, and so on. The first guy usually just wakes up to "acknowledge" that someone is taking care of the issue, and writes the wrap-ups and closes the channel.

rob74 2y ago

> If a resolution requires a system change, what organization would allow one person at 3am to make it and push to prod? Well yeah, if you have on call, I guess you have to trust the on-call person to make that decision. If they get called at 3am, the problem is obviously so bad that almost anything they do can only make it better, right? If it's some overlooked edge case that crashes the whole application and can be fixed with a one line change, it's probably Ok to do it a 3am. If it's more complicated, maybe wait until 7am and discuss it with others too. And, in any case, a thorough post mortem (during office hours) and checking if the 3am fix is really a proper fix for the issue is in order...

jjice 2y ago

The way I was taught to be on call by a guy I worked with that was also an SRE at a large software company was to "patch the hole in the tire and get it to the service center". It wasn't about fixing the problem or fully understanding it, but instead making sure the system can run for the time being, get some sleep, and have a more complete triage in the morning. I've found this to work pretty well the past few years. While I do agree with your sentiment, I'd say that the perspective I've learned about being on call is a big different than the one you've experienced (which may be the more common one, I'm not sure). So this may end up being a cultural difference between on-call situations.

AdieuToLogic 2y ago
> The way I was taught to be on call by a guy I worked with that was also an SRE at a large software company was to "patch the hole in the tire and get it to the service center". That is a great philosophy to have when supporting a prod system off-hours, one which I fully subscribe. > While I do agree with your sentiment, I'd say that the perspective I've learned about being on call is a big[sic] different than the one you've experienced (which may be the more common one, I'm not sure). My underlying thesis is not with being on-call, but instead expecting a developer to perform their non-support duties in addition to being on-call. The worst case scenario of this is when the on-call developer is also tier one support. If an organization wants to have developers perform SRE duties, presumably due to deeper understanding of a system, fine. Assign them to support and suspend development responsibilities during same. Just my $0.02
asdf6969 2y ago
In my experience we patch the hole in the tire and then get directed to patch a hole in another tire. Manager says work on the highest priority tire so now there are 2 partially broken tires but the problem is technically fixed. Repeat for 10 years and now it’s time to rebuild it “correctly”, but actually we just have two cars to patch since nobody knows what awful hacks we need to port over for “expected” behaviour

jyounker 2y ago

Devs need to be on call at three AM so that they suffer when their software fails. This is how you align developer motivation with operational motivation. The organization needs to incur financial penalties when the on-call staff have to respond at three AM. This aligns the organizations motivations with the operational motivations. When managers see an on-call-incident line-item going up then they're more willing to tell product to take a hike.

AdieuToLogic 2y ago
> Devs need to be on call at three AM so that they suffer when their software fails. This is how you align developer motivation with operational motivation. As I mentioned in a peer response: Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0]. 0 - https://nielsbohrmann.com/prices-law/ https://nielsbohrmann.com/prices-law/
- jyounker 2y ago
  It's interesting that you see this as being a punitive policy. Our job is producing quality systems. If our software is so bug ridden that caring for it is a punishment, then failed we've at producing quality software.