4 ms·
So I've seen personally been involved in this at both Google and Facebook. They have very different approaches. One works reasonably well. One... doesn't. At G
by cletus 5y ago
So I've seen personally been involved in this at both Google and Facebook. They have very different approaches. One works reasonably well. One... doesn't.
At Google, when you develop a service, you as a software team are responsible for maintaining it. This includes being oncall. But, depending on how critical the service is, you'll get bonus pay for this oncall time. This can amount to >$10k a year. And your oncall may not be that noisy. That is something and (IMHO) significant. Generally you'll be oncall for a week but this can vary.
For sufficiently high profile services, oncall may end up being owned by an SRE team. That's not really something that can be thrown over the fence by SWE teams. SRE teams have to accept that ownership. To do this, it requires meeting standards like having an oncall runbook, a sufficiently long history (~6 months), adequate metrics and so on. At that point, SWEs will still be second level support for something SREs may need help resolving. You'll still get paid for this.
SREs don't generally have week long oncalls. For the highest profile services, SRE support is global, meaning you'll have team members in 3 time zones such that whoever is oncall is in their normal working hours. So you might have an 8 hour oncall every week or something.
At Facebook, generally as a software team you'll be responsible for that service forever. Some key services may be fully or partially supported by PEs (Production Engineers). PEs are less common than Google SREs.
You don't get oncall pay at Facebook. It's simply expected. Depending on your management, you may be expected to do that oncall and everything else you're supposed to be doing.
So here's why Google's system is better than Facebook's:
1. Oncalls, in my experience, tend to be a lot less noisy at Google. Services tend to be much more mature and stable at Google than Facebook;
2. Teams are generally larger at Google. This means that not everyone needs to be oncall. This is good. There is an optimum size for oncalls. IMHO it is about 8-12. Fewer than 8 and people are oncall too much. More than 12 and you lose skills by not using them often enough.
3. The pay is really important to (2) because it means that those who are oncall at least get compensated for it. At a minimum this sends the message that oncall is important and rewarded. It also means those oncall are less likely to resent those not oncall, which might well be the case if you're not compensated for the extra work;
4. There are a ton of orphan projects at Facebook (IME). By this I mean some product team (in particular) will develop something, ship it and then... forget about it. Onwership of source code trees and projects tends to be fairly strictly enforced at Google. These orphan projects tend to create a number of support issues and tasks that are a drain on oncall as there is no one to pass those tasks on to.
5. Culturally, Google seems to acknowledge and respect oncall work load more than Facebook does. For example, tasks at Facebook have SLAs and I saw plenty of games played to avoid dealing with tasks. Examples: changing priority to increase or remove the SLA, silently closing tasks, passing tasks back months later to the reporter asking "is this still an issue?", etc.
So my point is that not wanting to be oncall is understandable but for many engineers, it's going to be part of your job. If so, you need to make sure your organization does it well. There's a world of difference between an oncall you don't get paid for that's 40 hours of work vs one where you get 2 tasks a week and you're getting paid for possibly having to answer an alert.
Hating oncall is indicative of a bad engineering culture IME. Things like prioritizing shipping above all else, not rewarding fixing things, unreasonable management driven deadlines and so on. Oncall is the canary in the coal mine for all of that.
- cghendrix 5y agoI have a question. Is this for all environments (dev, qa, staging or whatever the equivalent is at google/fb) or just production?
- joshuamorton 5y agoSREs are (with a few exceptions) only responsible for production (or more precisely, things serving prod traffic, which may be multiple environments). Dev teams are usually responsible for keeping qa and ci green.
- supernovae 5y agoDo you have kids? hobbies? friends? Do you like to go fishing on weekends? Do you like to go out on a date on weeknights? Do you like going to visit family on Christmas and Thanksgiving and leaving your laptop at home? It's great explaining to your 11 year old that you can't go to their recital because your on call isn't it? It boggles my mind that you suggest people lose skills by not using them often enough... these on call rotations aren't skills. What's weird, is that we default to look at the goodness or badness of something purely on "skills" and "money". Yes.. yes.. we've all read the SRE book and understand Googles approach to flexibility - but... it doesn't have to be that way. Ask yourself.. what's next? What are you looking to do after 25 years of being on call? Will you still enjoy it? I'll never have those nights and weekends back that I spent being on call for products and services that don't exist anymore. Sure, I made some money - but it didn't have to be at such a great cost.
- vitus 5y agoAt least at Google, oncall shifts aren't set in stone. If you need to swap with a coworker (even for a few hours), you can. Devs should not be holding 15-minute (much less 5-minute) pagers; if your system is really that critical, you should be working to onboard an SRE team. > these on call rotations aren't skills. Troubleshooting live production issues is a skill. Triaging incidents to prioritize actions to minimize user harm is a skill. Assessing tradeoffs of mitigation strategies is a skill. The very tangible scenario of being woken up in the middle of the night builds empathy with your devops / SRE organization (which can manifest itself in designing more resilient systems, engaging in less risky behavior, improving documentation for the next oncaller, etc). Oncall isn't for everyone, and it's certainly not worthwhile if you don't take away any lessons from it. But devs are ultimately the closest ones to their code, and they should take advantage of the opportunity to understand how it behaves in production. > Ask yourself.. what's next? What are you looking to do after 25 years of being on call? Will you still enjoy it? I've had bad weeks of oncall as well as quiet weeks. If it ever gets to a point where bad weeks are much more frequent, and nobody's doing anything about it, then yeah, I won't want to do it anymore. But as long as it's still providing value beyond humans providing blood / sweat / tears, and as long as we're actually trying to make things better, I don't mind holding a pager a few weeks a year.