6 ms·
I would respectfully say that you are wrong. I speak from experience. At Netflix we tried to hire for around the clock coverage. But what ended up working mu
by jedberg 5y ago
I would respectfully say that you are wrong. I speak from experience. At Netflix we tried to hire for around the clock coverage. But what ended up working much better was taking that same team and having each person on call for a week at a time, all based in Pacific Time.
Yes, you would get calls at 2am, sometimes multiple days in a row. But you were only on call once every six to eight weeks, and we scheduled out well in advance so you could plan your life accordingly.
As a bonus, for the five weeks you weren't on call, you were highly incentivized (and had the time) to build tools or submit patches to fix the problems that woke you at 2am.
> It requires (oh, the horror!) actually maintaining production standards, runbooks, and other documentation.
I disagree with this too. Documentation and runbooks are useless in an outage. Instead of runbooks, write code to do the thing. Instead of documentation, comment the code and build automation to make the documentation unnecessary, or at least surface the right information automatically if you can't automate it.
- magicalhippo 5y ago> But what ended up working much better was taking that same team and having each person on call for a week at a time, all based in Pacific Time. Our support team does the same, and they seem to be quite happy with it. They also get the following Friday off (in addition to compensation). They do their best to shield us developers from after-hour calls, usually one can get things moving enough that it can be handled properly in the morning.
- d0gsg0w00f 5y agoEven as a dedicated operations team for a product, we did this too. On call person worked tickets and took calls for one week at a time, the rest of the team worked on ways to make on-call suck less. For an eight person team it worked well for about three years until bigger stuff happened in the org and we all parted ways.
- solatic 5y ago> you were highly incentivized (and had the time) to build tools or submit patches to fix the problems that woke you at 2am. Ah, so you worked on a team where the SRE needs were prioritized over the feature requests? Because in most companies where I've worked, Product + Customer Service + Sales + Marketing + Executives don't really have time or patience for the engineers to get their diamond polishing cloths out. They want to see feature development. They're willing to be forced to prioritize exactly which feature they'll get soonest, and they understand that engineering needs time to keep the systems running, but in most businesses I've worked, the business comes first. > Documentation and runbooks are useless in an outage. Instead of runbooks, write code to do the thing. Instead of documentation, comment the code and build automation to make the documentation unnecessary We do that too. If you could write code to Solve All The Problems then you'd never need to page a human in the first place ;) I'll give you a simple example of where you can't write code to solve this sort of thing. Let's say that you have an autoscaler that will scale your server group up to X servers. You define an alert to page you if the autoscaler hits the maximum. The page goes off. Do you really want to write code to arbitrarily increase the autoscaler maximum whenever it hits the maximum? Why do you have the maximum in the first place? The entire reason why the autoscaler maximum exists is to prevent cost overruns from autoscaling run amok. You want a human being, not code, to look at the autoscaler and make the decision. Do you have steady-slow growth up to the maximum? Maybe it should be raised, if it represents natural growth. Maybe it shouldn't, if you just raised it last week and it shouldn't be anywhere near this busy. Do you have hockey-stick growth? Maybe the maximum is working as expected, looks like a resource leak hit production. Or maybe you have a massive traffic hit and you actually do want to increase the maximum. Maybe you'd prefer to take the outage from the traffic hit, let the 429s cool everyone off. But good luck trying to write code to handle that automatically, and correctly for you! > or at least surface the right information automatically if you can't automate it. Ah, well, that's exactly what the dedicated operations staff are doing, because when you have three follow-the-sun teams, you need standards, not three sets of people who each somehow telepathically share the same tribal knowledge? Don't get me wrong, I'm not anti-automation or something. If your operations folks are click-clicking in consoles all day long, the same click-clicking every day, probably something's wrong. But the SRE model asks for operations automation to stick within operations teams, not development teams.
- jedberg 5y ago> Ah, so you worked on a team where the SRE needs were prioritized over the feature requests? Yes, it was an SRE team. All we do is write tools to make operations better, but more importantly we write tools to make it easier for the dev teams to operate their own systems better. But yes, we had products teams that would push back on our requests because they had product to deliver, and that was fine. We'd either figure out how to do the work for them, or figure out a workaround. > We do that too. If you could write code to Solve All The Problems then you'd never need to page a human in the first place ;) Well yes, that's the idea. You can't get to 5 9s of reliability unless it's all automated. :) > I'll give you a simple example of where you can't write code to solve this sort of thing. I could easily write code to solve the thing. Step one, double the limit to alleviate immediate customer pain. Step two, page someone to wake up and look at the graphs and figure out what the better medium term solution is to get us through until the morning, including links to said relevant graphs. You're not gonna have a cost overrun doubling the limit for one night. And if there is a big problem, the person will get paged again a few hours later and have more information to make a better decision. > But the SRE model asks for operations automation to stick within operations teams, not development teams. Yes, but I'm not sure I see why that's bad. I don't see any purpose for a dedicated operations team, especially a follow the sun team. If you're Google and you already have offices all around the world, sure, it will be better. But it makes no sense to hire an around the world team just for operations if the rest of your company is in one time zone.
- aij 5y ago> Step one, double the limit to alleviate immediate customer pain. I've been oncall for systems where that would not work. Doubling the memory means you need twice as many machines. Depending on the service, that could require significantly increased network bandwidth. Now the network is saturated and every node needs to queue more data. Now latency and throughput are even worse, and even more requests are being dropped, so you automatically double the limit again...
- jedberg 5y ago
- deanc 5y agoThis is the same approach as night shifts for nurses. There’s a lot of evidence to suggest that the effects on this infrequent but consistent disturbance to their circadian rhythms causes all kinds of physiological damage. One example [1]. We have to do better. I think the original suggestion of finding specialised night workers or those in other timezones is more humane. [1] https://blogs.cdc.gov/niosh-science-blog/2021/04/27/nightshift-cancer/ https://blogs.cdc.gov/niosh-science-blog/2021/04/27/nightshi...
- jedberg 5y agoThat article is about night shift work, not day shift work that occasionally makes you work an hour or two at night every six weeks.
- toomuchtodo 5y agoHere is a reference that is a bit more attributable to the on call experience. There is a tangible human cost to after hours responses during an on call rotation. I personally do not recommend on call roles to any technology professional who can avoid them due to these health consequences of an on call requirement. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5449130/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5449130/ > Sleep plays a vital role in brain function and systemic physiology across many body systems. Problems with sleep are widely prevalent and include deficits in quantity and quality of sleep; sleep problems that impact the continuity of sleep are collectively referred to as sleep disruptions. Numerous factors contribute to sleep disruption, ranging from lifestyle and environmental factors to sleep disorders and other medical conditions. Sleep disruptions have substantial adverse short- and long-term health consequences. A literature search was conducted to provide a nonsystematic review of these health consequences (this review was designed to be nonsystematic to better focus on the topics of interest due to the myriad parameters affected by sleep). Sleep disruption is associated with increased activity of the sympathetic nervous system and hypothalamic–pituitary–adrenal axis, metabolic effects, changes in circadian rhythms, and proinflammatory responses. In otherwise healthy adults, short-term consequences of sleep disruption include increased stress responsivity, somatic pain, reduced quality of life, emotional distress and mood disorders, and cognitive, memory, and performance deficits. For adolescents, psychosocial health, school performance, and risk-taking behaviors are impacted by sleep disruption. Behavioral problems and cognitive functioning are associated with sleep disruption in children. Long-term consequences of sleep disruption in otherwise healthy individuals include hypertension, dyslipidemia, cardiovascular disease, weight-related issues, metabolic syndrome, type 2 diabetes mellitus, and colorectal cancer. All-cause mortality is also increased in men with sleep disturbances. For those with underlying medical conditions, sleep disruption may diminish the health-related quality of life of children and adolescents and may worsen the severity of common gastrointestinal disorders. As a result of the potential consequences of sleep disruption, health care professionals should be cognizant of how managing underlying medical conditions may help to optimize sleep continuity and consider prescribing interventions that minimize sleep disruption.
- uselessextras 5y agoI agree with you completely, especially on the last paragraph. No pain - no gain.