6 ms·

Tips for on Call Engineers During the Holidays

28 points by kiyanwang 6y ago

howlgarnish 6y ago
Oh, how I don't miss being responsible for building out the holiday oncall schedule. A few more tips: - Lead by example. If you're responsible for building the schedule, do a shift yourself, and don't cherry pick the least bad one either. - Remember last year's schedule: it's shitty enough to do one Christmas, it's worse to do two in a row. - But also avoid the trap of scheduling only noobs on the worst shifts. At least the secondary should be a grizzled veteran. - Have an escalation plan in place if shit really hits the fan. Remember the Lizard Squad attacks during Christmas 2014? Good times: https://en.wikipedia.org/wiki/Lizard_Squad https://en.wikipedia.org/wiki/Lizard_Squad
- x0x0 6y ago
  The first is the most crucial. Execs and managers sign up to take shifts.
- spoonjim 6y ago
  With the numbers of Indians and Chinese in Silicon Valley is this really a problem? I'd imagine that there are plenty of engineers who couldn't care less about Christmas.
  - jedberg 6y ago
    Christmas is a most secular holiday. Everyone gets together with family for Christmas. Even us Jews.
  - howlgarnish 6y ago
    Yes, it's still a problem. New Year's Eve around midnight in particular is a shift that almost nobody wants to take.
sfkdjf9j3j 6y ago
> To reduce the pain further, consider upping on call bonuses for the entire week Are you guys getting bonuses for being on call? I kind of signed up for it (SRE) but I'm curious if this is more common in jobs that aren't primarily concerned with reliability.
- kache_ 6y ago
  I get paid for oncall. Why would I do it otherwise?
  - sfkdjf9j3j 6y ago
    Most places with on call shifts (that I've worked for at least - VC funded tech companies) treat it as a responsibility like any other for a salaried worker. I'm sure individuals have quietly negotiated special deals as they tend to do, but it's definitely not a common practice to pay extra for it specifically.
    - kache_ 6y ago
      I guess it really varies from system to system. The one I work on has pretty strict uptime requirements so we get well compensated for it.
    - hotsauceror 6y ago
      Of my last three employers, one had a dept/team policy of a weekly on-call bonus. The on-call person got a few hundred bucks extra (taxed, but still...), and there was a budget line item for it. It was a pretty nice token of appreciation. The other two - one had an on-call rotation no extra bonus but an informal comp time policy if you had a bad night. The other had nothing. The expectation was that the two guys on the team were on-call all the time. "Oh, we'll probably never call you." Ironically, of the three, the one with the on-call bonus budget was the one where you almost never got called.
    - howlgarnish 6y ago
      There are different tiers of oncall. Needing to be online and actively debugging within 5 minutes of getting paged is far more constraining than being the third line of internal escalation that gets a phone call maybe twice a year.
alfalfasprout 6y ago
The best one is companywide code freeze well before the actual holidays (minimum 3-4 days before). That gives you 3-4 days to discover a whole slew of possibly nasty things and revert them before the week happens. This keeps the likelihood of anything actually bad happening during the holidays way, way down.
- tdons 6y ago
  However... it can also trigger new behaviour because programs / services run for longer than normal because deployments aren't being refreshed as frequently as normal.
  - TheDong 6y ago
    This seems like it's just being contrarian for the point of it, even if it doesn't lead to any interesting conversation. I get it, we're hackers, we like to think of edge cases and are pedantic as hell. But pedantry and edge cases aren't always useful in conversation.
    - danielheath 6y ago
      This caused at least one outage in my career. Memory leak caused every single app server to crash in the space of a few hours on new years eve; regular deploys had been hiding it.
    - Hamuko 6y ago
      I don't think it's even that much of an edge case. We've definitely had memory leaks on all sorts of services just because they were running longer than usual without a deployment.
    - Schnitz 6y ago
      This is unfortunately all too common, but it can at least be solved with a simple restart - and truly fixed after the holidays, which is much better than a 4h debug fest because of a typo in some code someone wanted to quickly get in before the holidays.
vvoyer 6y ago
Hey HN, shameless plug: I created a Slack app to help teams create on-call shift schedules: https://turnshift.app https://turnshift.app. 1. You create a schedule 2. Members get messages to book shifts 3. Notifications are sent to channels about who’s on shift We’ve used this technique at Algolia to organize the whole-company support (40 engineers) and in general share any repetitive team task automatically. My tool is very new so really looking for feedback. If this gets some love I’ll create a proper HN post.

28 points by kiyanwang 6y ago

Oh, how I don't miss being responsible for building out the holiday oncall schedule. A few more tips: - Lead by example. If you're responsible for building the schedule, do a shift yourself, and don't cherry pick the least bad one either. - Remember last year's schedule: it's shitty enough to do one Christmas, it's worse to do two in a row. - But also avoid the trap of scheduling only noobs on the worst shifts. At least the secondary should be a grizzled veteran. - Have an escalation plan in place if shit really hits the fan. Remember the Lizard Squad attacks during Christmas 2014? Good times: https://en.wikipedia.org/wiki/Lizard_Squad https://en.wikipedia.org/wiki/Lizard_Squad

x0x0 6y ago
The first is the most crucial. Execs and managers sign up to take shifts.
spoonjim 6y ago
With the numbers of Indians and Chinese in Silicon Valley is this really a problem? I'd imagine that there are plenty of engineers who couldn't care less about Christmas.
- jedberg 6y ago
  Christmas is a most secular holiday. Everyone gets together with family for Christmas. Even us Jews.
- howlgarnish 6y ago
  Yes, it's still a problem. New Year's Eve around midnight in particular is a shift that almost nobody wants to take.

sfkdjf9j3j 6y ago

> To reduce the pain further, consider upping on call bonuses for the entire week Are you guys getting bonuses for being on call? I kind of signed up for it (SRE) but I'm curious if this is more common in jobs that aren't primarily concerned with reliability.

kache_ 6y ago
I get paid for oncall. Why would I do it otherwise?
- sfkdjf9j3j 6y ago
  Most places with on call shifts (that I've worked for at least - VC funded tech companies) treat it as a responsibility like any other for a salaried worker. I'm sure individuals have quietly negotiated special deals as they tend to do, but it's definitely not a common practice to pay extra for it specifically.
  - kache_ 6y ago
    I guess it really varies from system to system. The one I work on has pretty strict uptime requirements so we get well compensated for it.
  - hotsauceror 6y ago
    Of my last three employers, one had a dept/team policy of a weekly on-call bonus. The on-call person got a few hundred bucks extra (taxed, but still...), and there was a budget line item for it. It was a pretty nice token of appreciation. The other two - one had an on-call rotation no extra bonus but an informal comp time policy if you had a bad night. The other had nothing. The expectation was that the two guys on the team were on-call all the time. "Oh, we'll probably never call you." Ironically, of the three, the one with the on-call bonus budget was the one where you almost never got called.
  - howlgarnish 6y ago
    There are different tiers of oncall. Needing to be online and actively debugging within 5 minutes of getting paged is far more constraining than being the third line of internal escalation that gets a phone call maybe twice a year.

alfalfasprout 6y ago

The best one is companywide code freeze well before the actual holidays (minimum 3-4 days before). That gives you 3-4 days to discover a whole slew of possibly nasty things and revert them before the week happens. This keeps the likelihood of anything actually bad happening during the holidays way, way down.

tdons 6y ago
However... it can also trigger new behaviour because programs / services run for longer than normal because deployments aren't being refreshed as frequently as normal.
- TheDong 6y ago
  This seems like it's just being contrarian for the point of it, even if it doesn't lead to any interesting conversation. I get it, we're hackers, we like to think of edge cases and are pedantic as hell. But pedantry and edge cases aren't always useful in conversation.
  - danielheath 6y ago
    This caused at least one outage in my career. Memory leak caused every single app server to crash in the space of a few hours on new years eve; regular deploys had been hiding it.
  - Hamuko 6y ago
    I don't think it's even that much of an edge case. We've definitely had memory leaks on all sorts of services just because they were running longer than usual without a deployment.
  - Schnitz 6y ago
    This is unfortunately all too common, but it can at least be solved with a simple restart - and truly fixed after the holidays, which is much better than a 4h debug fest because of a typo in some code someone wanted to quickly get in before the holidays.

vvoyer 6y ago

Hey HN, shameless plug: I created a Slack app to help teams create on-call shift schedules: https://turnshift.app https://turnshift.app. 1. You create a schedule 2. Members get messages to book shifts 3. Notifications are sent to channels about who’s on shift We’ve used this technique at Algolia to organize the whole-company support (40 engineers) and in general share any repetitive team task automatically. My tool is very new so really looking for feedback. If this gets some love I’ll create a proper HN post.