6 ms·

What do you find exhausting? One anti-pattern I've found is that most orgs ask a single team to handle on-call around the clock for their service. This rarely

• deleted 5y ago
  [deleted]
• grogers 5y ago
  The solution to get paged at off hours a lot is rarely to hire additional teams to cover those times for you, at least not long term. For things you can control, you should fix the root causes of those issues. For things you can't control you should spend effort on making them within your control (eg architecture improvement). This takes time, so follow-the-sun rotation might be a stop gap solution, but you need to make sure it doesn't cover over the real problems without them getting any better.
  • ahelwer 5y ago
    From experience, it's really hard to fix the root causes of issues when you were woken up three times the night before and had two more of the same incident occur during the workday. In my case I struggled along for a couple years but the best thing to do was just leave and let it be someone else's problem.
    • kqr 5y ago
      Best thing for what? Surely not software quality and customer satisfaction.
      • ahelwer 5y ago
        If they cared about that they would either pay me so much money I'd be insane to walk away or they would hire people in other time zones to cover the load. Instead they chose to pay for their customer satisfaction with my burnout. The thing about that strategy is... eventually the thing holding their customer satisfaction together gets burnt out. So I leave. And even then they're still getting the better half of the bargain.
        • kqr 5y ago
          Sorry, I accidentally said you did the wrong thing for leaving. That wasn't my intention. Of course, leaving was the right choice for you. What I meant was the company you were working for does not get the best quality or customer satisfaction by overworking you to the point where you have to leave. It would have been better for their software quality to handle things differently.
• notacoward 5y ago
  This. Absolutely this. Working on large distributed system can be both exhilarating and exhausting. The two often go hand in hand. However, working on such systems without diligence tips the scales toward exhausting. If your testing and your documentation and your communication (both internal and with consumers) suck, you're in for a world of pain. "But writing documentation is a waste of time because the code evolves so fast." Yeah, I hear that, but there's also a lot of time lost to people harried during their on-call and still exhausted for a week afterward, to training new people because the old ones burned out or just left for greener pastures, to maintaining old failed experiments because customers (perhaps at your insistence) still rely on them and backing them out would be almost as much work than adding them was, and so on. That's not really moving fast. That's just flailing. You can actually go further faster if you maintain a bit of discipline. Yes, there will still be some "wasted" time, but it'll be a bounded, controlled waste like the ablative tiles on a re-entry vehicle - not the uncontrolled explosion of complexity and effort that seems common in many of the younger orgs building/maintaining such systems nowadays.
  • bckr 5y ago
    > That's not really moving fast. That's just flailing. Yes, a million times yes. This is moving me. Where do I find a team that understands this wisdom?
• Ozzie_osman 5y ago
  I'd argue that timezone is just part of the problem. If you're responsible for a high oncall load, you are subjected to a steady, unpredictable stream of interrupts requiring you to act to minimize downtime or degradation. Obviously it's worse if you get these at night, but it's still bad during the day. I think the anti-pattern is having one team responsible for another's burden. You want teams to both be responsible for fixing their own systems when they break, AND be empowered to build/fix their broken systems to minimize oncall incidents.
• NationalPark 5y ago
  I don’t think this is a stable long term solution. The “on call” teams end up frustrated with the engineers who ship bugs and this results in added process that delays deploys, arbitrary demands for test coverage, capricious error budgets, etc. It’s much better to have the engineers who wrote the code be responsible for running it, and if their operational burden becomes too high, to staff up the dev team to empower them to go after root causes. Plus the engineers who wrote the code always have better context than reliability people who tend to be systems experts but lack the business logic intuition to spot errors at a glance.
  • kqr 5y ago
    Correct. Throwing software over the wall to "other people" and letting them deal with the problems of running the software is guaranteed to lead to low quality, inefficient processes, or usually both.
  • Hermitian909 5y ago
    I don't think the parent was implying you're never on call for your code, just only on call during working hours. One of the challenges for larger companies in trying to make teams on-call 24/7 is that your most senior engineers often have enough money that they don't have to take on-call. Some variation of the following conversation happens in Big Tech more than most people seem to anticipate: "hey, so I have 7 mil in the bank, a house, and kids; so I'm not taking on-call anymore" "I understand on-call is a burden, but the practice is a big part of how we maintain operational excellence" "Alright, I quit" "Woah woah woah, uh, ok, what about we work on transitioning you out of on call over the next 6 months?" "Nah, I'm done" "This is going to be really disruptive to the team!" "Yeah man it sucks, I really feel for you" My understanding is a few famous outages at large cloud providers are a direct result of management not anticipating these conversations and assuming 24/7 on-call from a single geographically centered team of high powered engineers was sustainable.
    • deleted 5y ago
      [deleted]
  • solatic 5y ago
• toast0 5y ago
  At the end of the day, there's a human cost to responding to pages, and there's a human cost to collaboration. Both of those can drive burn out. Personally, I find all that collaboration work very hard and stressful, so I work better in a situation where I get pages for the services I control; but that would change if pages were frequent and mostly related to dependencies outside of my control. It also helps to have been working in organizations that prioritize a working service over features. Getting frequent overnight issues that can't be resolved without third party effort that's not going to happen anytime soon is a major problem that I see reports of in threads like this. I can also get behind a team that can manage the base operations issues like ram/storage/cpu faults on nodes and networking. The runbooks for handling those issues are usually pretty short and don't need much collaboration.
• jedberg 5y ago
  I would respectfully say that you are wrong. I speak from experience. At Netflix we tried to hire for around the clock coverage. But what ended up working much better was taking that same team and having each person on call for a week at a time, all based in Pacific Time. Yes, you would get calls at 2am, sometimes multiple days in a row. But you were only on call once every six to eight weeks, and we scheduled out well in advance so you could plan your life accordingly. As a bonus, for the five weeks you weren't on call, you were highly incentivized (and had the time) to build tools or submit patches to fix the problems that woke you at 2am. > It requires (oh, the horror!) actually maintaining production standards, runbooks, and other documentation. I disagree with this too. Documentation and runbooks are useless in an outage. Instead of runbooks, write code to do the thing. Instead of documentation, comment the code and build automation to make the documentation unnecessary, or at least surface the right information automatically if you can't automate it.
  • magicalhippo 5y ago
    > But what ended up working much better was taking that same team and having each person on call for a week at a time, all based in Pacific Time. Our support team does the same, and they seem to be quite happy with it. They also get the following Friday off (in addition to compensation). They do their best to shield us developers from after-hour calls, usually one can get things moving enough that it can be handled properly in the morning.
  • d0gsg0w00f 5y ago
    Even as a dedicated operations team for a product, we did this too. On call person worked tickets and took calls for one week at a time, the rest of the team worked on ways to make on-call suck less. For an eight person team it worked well for about three years until bigger stuff happened in the org and we all parted ways.
  • solatic 5y ago
    > you were highly incentivized (and had the time) to build tools or submit patches to fix the problems that woke you at 2am. Ah, so you worked on a team where the SRE needs were prioritized over the feature requests? Because in most companies where I've worked, Product + Customer Service + Sales + Marketing + Executives don't really have time or patience for the engineers to get their diamond polishing cloths out. They want to see feature development. They're willing to be forced to prioritize exactly which feature they'll get soonest, and they understand that engineering needs time to keep the systems running, but in most businesses I've worked, the business comes first. > Documentation and runbooks are useless in an outage. Instead of runbooks, write code to do the thing. Instead of documentation, comment the code and build automation to make the documentation unnecessary We do that too. If you could write code to Solve All The Problems then you'd never need to page a human in the first place ;) I'll give you a simple example of where you can't write code to solve this sort of thing. Let's say that you have an autoscaler that will scale your server group up to X servers. You define an alert to page you if the autoscaler hits the maximum. The page goes off. Do you really want to write code to arbitrarily increase the autoscaler maximum whenever it hits the maximum? Why do you have the maximum in the first place? The entire reason why the autoscaler maximum exists is to prevent cost overruns from autoscaling run amok. You want a human being, not code, to look at the autoscaler and make the decision. Do you have steady-slow growth up to the maximum? Maybe it should be raised, if it represents natural growth. Maybe it shouldn't, if you just raised it last week and it shouldn't be anywhere near this busy. Do you have hockey-stick growth? Maybe the maximum is working as expected, looks like a resource leak hit production. Or maybe you have a massive traffic hit and you actually do want to increase the maximum. Maybe you'd prefer to take the outage from the traffic hit, let the 429s cool everyone off. But good luck trying to write code to handle that automatically, and correctly for you! > or at least surface the right information automatically if you can't automate it. Ah, well, that's exactly what the dedicated operations staff are doing, because when you have three follow-the-sun teams, you need standards, not three sets of people who each somehow telepathically share the same tribal knowledge? Don't get me wrong, I'm not anti-automation or something. If your operations folks are click-clicking in consoles all day long, the same click-clicking every day, probably something's wrong. But the SRE model asks for operations automation to stick within operations teams, not development teams.