6 ms·

Ask HN: Does your on-call rotation suck? Can I join it?

• tra3 1y ago
  You know it's broken, everyone knows it's broken but it keeps alerting.
  • andrewmcwatters 1y ago
    And you're allocated the time for on-call, salaried for it, but never assigned the task to fix the damn thing. So instead, you and your team burn maybe-I-have-to-get-up-at-night-time instead of definitely-attempting-to-fix-it-time.
    • tra3 1y ago
      It's like hitting snooze on the alarm clock. It's an illusion. You gonna have to wake up. You're not getting any more sleep. I stopped hitting snooze as I got older. I either wake up or give up and sleep in..
    • asciifree 1y ago
      I strongly believe whoever is on-call should have free reign to modify anything about their ops environment. The pitch for management is that while in the short term it might take time away from project work, eventually the reduction in interruptions will result in higher productivity. p.s. Knew I recognised the name - loved following development of Planimeter/Grid a while ago!
      • andrewmcwatters 1y ago
        Thank you so much! My company is shipping a finance platform at the moment, and I’d love to get back to Planimeter’s work when I am able.
  • nik736 1y ago
    Yes, this is probably the #1 reason. Alerts go off, but you don't even know if they are for real issues this time.
• johncole 1y ago
  Doing some customer discovery?
• Quitschquat 1y ago
  Ask the poor bastards at Discovery
• brudgers 1y ago
  On-call issues are staffing issues not tools issues. Inadequate staff is the only reason on-call exists. Sure, people might be mostly sitting around all night being paid and not being terribly busy. But if a company needs someone at night, they need someone at night. Companies getting away with not paying for that is why oncall sucks. In other words oncall sucks because companies don’t pay for solving the problems that require it. There’s no self correcting feedback. A tool can’t fix that and oncall is not inevitable. Good luck.
  • al_borland 1y ago
    I used to work the night shift handling most off hours issues for a a couple dozen teams. We would occasionally have to call someone, but not that often compared to the alternative. Most of the time it was just to get sign off on what we already planned to do. When I started people were paid for any hours they worked on-call. By the end, the company changed the policy so on-call was part of base pay. For those who were on-call during the change over, their last year of on-call pay was averaged and added to their salary. For everyone who came after that, they got screwed (that includes me). Once I changed to the day shift I got called a few times for on-call. Every single time, I documented what I did to fix it, as I did it, and handed it off to the ops team. Or in some cases I automated the fix. I have 0 tolerance for being called in my free time. I don’t care what the boss says my priorities are, if I’m being called at night, stopping that in its tracks is my #1 priority. If I ever get called two times for the same issue, that’s my fault. So far, it’s never happened.
    • asciifree 1y ago
      > When I started people were paid for any hours they worked on-call I've yet to hear of any alternative compensation model that actually works. Just pay people in their choice of money or time off in lieu. Sorry to hear you got screwed. > Every single time, I documented what I did to fix it, as I did it, and handed it off to the ops team. Or in some cases I automated the fix. I have 0 tolerance for being called in my free time. I don’t care what the boss says my priorities are, if I’m being called at night, stopping that in its tracks is my #1 priority. 100% agree, I think people are far too tolerant of being paged. Especially management - the productivity impact of constant interrupts is huge. In a previous job one of my favourite things to do was go out to teams and just disable alerts they said were noisy or unactionable. If there was any pushback/consequence I was happy to accept responsibility (but never had to).
• dgunay 1y ago
  I'm certainly not at liberty to invite a random to cover on-call shifts for us, but here's some anecdata about things I've witness that made on-call suck. We began with free food delivery over the weekend, and the expectation that you'd take a day off the next week ("unlimited" PTO policy). Eventually they stopped letting us do that and now the "unlimited" in our PTO policy has an invisible limit, so you can't actually do that without it counting towards the invisible limit on your unlimited PTO for the year. Our monitoring and alerting is unusably noisy. Deviance is fully normalized. All our postmortems typically have a section stating that alerts were issued, but ignored until customers began complaining. Attempts to cut the noise down to a sane level have all been defeated by the ever present pressure to feature factory. TBF this is mostly an engineering self-own and I feel partially responsible for this outcome. The on-call engineer does a shocking amount of manual labor to paper over bugs in the product and un-stick users who fall through the (many) cracks. It is effectively a T3 tech support rotation. We've taken steps to tone it down to mere triage and channel this into pressure against offending teams' timelines, but there's a huge amount of silent cultural resistance and no one is being held accountable when a feature increases support load. I suspect this issue alone would make most bigtech engineers quit. For the (many) issues that require manual intervention, the on-call engineer cannot actually do anything unless 2 other engineers sign off on a PR (either to run a SQL query or to deploy some tool or bugfix to resolve the problem). This is more specific to the product I work on, but the sheer amount of 3rd party services we rely on means that something is constantly acting up and there's not a lot we can do about it. Our API client code for each service we use typically contains _at least_ one service-specific hacky workaround to keep things running in the face of bad behavior. The frontend team has no on-call rotation despite causing plenty of bugs on their own. Backend engineers are expected to triage what are clearly frontend problems. We stood up a lot of observability tooling for the frontend but it took years for them to even start to use it. More than anything, it feels like the moment I stop championing the issue, everyone stops paying attention and the on-call experience reverts to the mean. Other on-call engineers just sort of stop boyscouting and let the chaos wash over them while focusing on sprint obligations (can't blame them), and leadership takes their eye off the ball to chase growth (also can't blame them). Hugely fucked lack of accountability and the buck eventually stops at whoever is the poor guy holding the pager that week.
  • 1y ago
• dakiol 1y ago
  On call sucks because: You need to stay at home or very close 24h. You cannot go for a run, or to the cinema, or to have a dinner if you are on call (please don’t tell me: “just take your laptop with you”. How on earth would I run with a laptop?) No matter if you get paged or not, you need to be available, and that sucks.
• joshstrange 1y ago
  “Hi, welcome to being on-call, watch these channels and respond to any alerts that fire, except those alerts, you can ignore this subset of alerts, also you can ignore this alert unless it fires multiple times in quick succession, and this alert only matters if….”
• ferguess_k 1y ago
  On call sucks because it's usually 7/24 for the on-call person. Any work that is 7/24 standby sucks anyway.
• moomoo11 1y ago
  PIP if you miss the 5 min threshold to respond at 3am.