3 ms·

If you've actually never been on call, it's hard to understand the way the body responds without experiencing it really. Having every other weekend cut short a

If you've actually never been on call, it's hard to understand the way the body responds without experiencing it really. Having every other weekend cut short as you always need to be near a PC from 4pm Sunday... The sheer terror of being woken up from a deep sleep after a long night of work just as you've finally fallen asleep... Resigning yourself to not sleeping as you finally close out the call, full of adrenaline as you see the first glimmers of light in the sky... Knowing the problem is the same problem as last week, last month, and that the causes are things outside your control (the 1 staffer in Asia being sick or on holiday, insufficient hardware CTO won't upgrade, bugs in orphaned upstream systems, political turf wars)... etc

tharkun__ 6y ago
It depends on the environment. I have been in both types. I have been in an environment for about 5 years where we Devs were on call for 7 days every month and a half-ish. This was with a business that saw traffic 24/7 with weekends and off hours being higher actually. I was called at 2am once and a couple times on a weekend and I didn't even have to open my computer. I was able to instruct the NOC in what a safe reboot procedure (that wasn't in their SOP) would be and it solved the issue. We had automated testing including performance testing with signoff required from the architect if we were too far off the baseline. This was an awesome environment to be working in and being on call was both easy money and if you ever did get a call you knew it was because something really went wrong. Contrast that with my current environment where you couldn't pay me enough money to make me sign up for the on call rotation. Monitors that fire Pager Duty alerts multie times a day including nights. An absolute nightmare and many if not most of these self heal too.
- steveBK123 6y ago
  Yes, 2 weekend & 1 night call in 5 years is absolutely tolerable. I was at a shop with clever management that fired faster than they backfilled and left the team in such disarray that the replacements started quitting as well. Ended up from a team of 6 down to effectively me & another guy rotating support. I’m talking about getting woken up once or twice a month at 2-3am to join a live incident bridge & debug systems for 2 hours, while being asked to give an update & ETA every 15 minutes, trying not to wake your wife.. and finishing just in time to see the light in the sky. To then head into the office and deal with more fire drills all day and then get asked why feature releases were behind schedule. Current org has auto-generating incident channels in slack which seems clever until you are on rotation and there are 5 different incident channels live at any given time with messaging popping in throughout the day as some are resolved in minutes, hours or days. A lot of this is just management failing the marshmallow test.

tharkun__ 6y ago

It depends on the environment. I have been in both types. I have been in an environment for about 5 years where we Devs were on call for 7 days every month and a half-ish. This was with a business that saw traffic 24/7 with weekends and off hours being higher actually. I was called at 2am once and a couple times on a weekend and I didn't even have to open my computer. I was able to instruct the NOC in what a safe reboot procedure (that wasn't in their SOP) would be and it solved the issue. We had automated testing including performance testing with signoff required from the architect if we were too far off the baseline. This was an awesome environment to be working in and being on call was both easy money and if you ever did get a call you knew it was because something really went wrong. Contrast that with my current environment where you couldn't pay me enough money to make me sign up for the on call rotation. Monitors that fire Pager Duty alerts multie times a day including nights. An absolute nightmare and many if not most of these self heal too.

steveBK123 6y ago
Yes, 2 weekend & 1 night call in 5 years is absolutely tolerable. I was at a shop with clever management that fired faster than they backfilled and left the team in such disarray that the replacements started quitting as well. Ended up from a team of 6 down to effectively me & another guy rotating support. I’m talking about getting woken up once or twice a month at 2-3am to join a live incident bridge & debug systems for 2 hours, while being asked to give an update & ETA every 15 minutes, trying not to wake your wife.. and finishing just in time to see the light in the sky. To then head into the office and deal with more fire drills all day and then get asked why feature releases were behind schedule. Current org has auto-generating incident channels in slack which seems clever until you are on rotation and there are 5 different incident channels live at any given time with messaging popping in throughout the day as some are resolved in minutes, hours or days. A lot of this is just management failing the marshmallow test.