9 ms·
Breaking Up with On-Call
- caseyohara 2y agoNot to distract from the article, but I'm pretty sure that photo of the guard tower is from Manzanar, one of the concentration camps in California that interned Japanese Americans during WWII. Probably not in great taste to use that photo to represent the idea of "guard duty" in software. https://en.wikipedia.org/wiki/Manzanar https://en.wikipedia.org/wiki/Manzanar
- idiotsecant 2y agoEh, intent matters. They probably just googled 'guard tower' and used the first one that looked close enough.
- caseyohara 2y agoThat’s my assumption and why I said something. I’m guessing they didn’t know.
- strken 2y agoApart from anything else, a fire tower is a better metaphor for on-call than a guard tower, too.
- abrookewood 2y ago[flagged]
- brigandish 2y agoWho, specifically, would that picture offend or change protect?
- Xylakant 2y agoPeople that have been interned in this or one of the other camps, or their descendants. It’s one generation ago, people born and raised in camps are still alive. George Takeo for example was born in one of the segregation camps. It’s a low stakes change. Nobody assumes harmful intentions from the author - I would not have recognized the picture either. But now that it’s been pointed out that it’s from a site where people were held illegally against their will, the reaction is a tell-tale. Knowing this, and insisting on keeping the images is now willfully associating with harmful behavior. Apart from that, I cannot associate with the picture either - as an on-call engineer for some widely used infrastructure, I am not a guard on duty keeping people in a camp. I am an emergency responder. I fix things when they go haywire or an accident happens. A firefighter, paramedics, civil emergency responder would IMO be a much better metaphor for what I do.
- modo_mario 2y ago>Nobody assumes harmful intentions from the author Then what's the issue or purpose? Other than satisfying or inflating some own moral selfimage? That kind of social dynamic appears to be what this kind of things is about most of the time as opposed to preventing mental harm or stopping such issues from (re)occurring. >Knowing this, and insisting on keeping the images is now willfully associating with harmful behavior. Do you not associate with harmfull behavior in far far more direct ways? Perhaps something that's considered ridiculous for most people to avoid like paying taxes that get spent on bombs or the like? If i want to make an article showing a prison and i pick one from google does it matter if that prison happened to be used in the past to house civil rights activists? Is the implication that that somehow normalises, advocates for or otherwise inches towards oppressing civil rights movements in any meaningfull way?
- dxdm 2y ago> Then what's the issue or purpose? To not use a thing that symbolizes the indiscriminate incarceration of innocent people as some clipart for an article about oncall. It's in bad taste, and also the wrong metaphor, and it sends the wrong message, even if not done on purpose. > Other than satisfying or inflating some own moral selfimage? Morality has a subjective component and often works by seeing through other people's eyes: How would I feel if this happened to me? How will other people see me if I did this? There's no universal consensus for this, of course, thankfully. So when people disagree about morality, I've found that it can be hard for the side that doesn't see a problem to understand what the other side is making such a fuss about. I also think there can be a performative aspect to moralizing. But I don't think it's fair or warranted to jump to the conclusion that just because one doesn't see the issue, that the other person must be posturing - because that's how you can make sense of their behavior from within your own point of view. People see things that rub them wrong, they speak up, as did you. How would you know you're not just inflating your self-image by putting down their comment like this? If you aren't, maybe they aren't, either.
- Xylakant 2y agoSo you're essentially saying: "I don't believe people should try to be nice to each other even in places where it essentially comes for free." I get it, it's impossible to do the right thing all of the time. It's a chore. But very often, it comes for essentially free. Help carry the baby prom down the stairs if the elevator is broken. Tell people their shoelace is open. Change a picture that symbolizes hurt to people in places where the point is not talking about these places. Call people by their chosen name. It comes for almost free. It makes other peoples life better. What's the argument against it?
- pokoleo 2y agoFrom my experience working on SaaS, and improving ops at large organizations, I've seen that "on-call culture" often exists inversely proportional to incentive alignment. When engineers bear the full financial consequences of 3AM pages, they're more likely to make systems more resilient by adding graceful failure modes. When incident response becomes an organizational checkbox divorced from financial outcomes and planning, you get perpetual firefighting. The most successful teams I've seen treat on-call like a leading indicator - every incident represents unpriced technical debt that should be systematically eliminated. Each alert becomes an investment opportunity rather than a burden to be rotated. Big companies aren't missing the resources to fix this; they just don't have the aligned incentive structures that make fixing it rational for individuals involved. The most rational thing to do as an individual on a bad rotation: quit or transfer.
- tacticus 2y agoIMO it's when the incident response and readiness practice imposes a direct backpressure on feature delivery that you get the issues actually fixed and a resilient system. if it's just the engineer while product and management see no real cost then people burn out and leave. > The most successful teams I've seen treat on-call like a leading indicator - every incident represents unpriced technical debt that should be systematically eliminated. Each alert becomes an investment opportunity rather than a burden to be rotated. 100%
- hinkley 2y agoThe people who do the real work don’t get raises and promotions because the annual review system punishes them for doing the right thing.
- rednafi 2y ago> When engineers bear the full financial consequences of 3AM pages, they're more likely to make systems more resilient by adding graceful failure modes. Making engineers handle 3 AM issues caused by their code is one thing, but making them bear the financial consequences is another. That’s how you create a blame-game culture where everyone is afraid to deploy at the end of the day or touch anything they don’t fully understand.
- stackskipton 2y agoI have no idea what article was trying to convey since on call was poorly defined I also couldn’t take it seriously when article opened with this. > Startups cannot afford engineers to baby sit software, big tech does. Say what? As ops person, I’ve seen multiple startups where devs are drowning in operational issues because software was written enough for feature MVP, ship with poor testing then constantly poke at with sticks to keep it working in hopes they could get enough revenue to not flame out.
- apeescape 2y agoWell, did those startups succeed? If not, I think it means the author was spot on in saying startups can't _afford_ that.
- stackskipton 2y agoOne did, two did not. None of due to engineering. Engineering rarely matters unfortunately but ability to weave a story and sell that story.
- apeescape 2y agoFair point about the importance of storytelling, but I wouldn't say "engineering rarely matters" in the tech industry. Compelling vision can mask those problems for a long while, but at some point the wheels will fall off if engineers can't deliver.
- jiveturkey 2y ago> I have no idea what article was trying to convey An ad for their consultancy. It's a call to action, not a balanced analysis.
- danpalmer 2y agoAs a current SRE, and having worked in a small startup, this doesn't echo my experience at all. What the author describes is possible what we would call "on duty" work, the grunt/maintenance work that comes with big software systems. It's not fun, and most companies/teams have friction getting this sort of work done. It's also however not how my SRE role is defined by any stretch. Our on-call work is much more about support during exceptional, somewhat rare circumstances.
- gpi 2y agoAgreed. Author's point of view seemed to have been influenced by their bad experience being part of the "on duty" work.
- strken 2y agoYeah, I was similarly confused. My experience has been that on-call is a roster of who will keep their laptop with them on the weekend. Incidents involve little to no customer comms, because if anything more than a sentence once an hour is necessary then there's someone else on call to handle it. Support work is a necessary nuisance, but it's also not what on-call is meant to be.
- coolgoose 2y agoSounds like normal support work not sure why that's affecting morale. It's normal in any kind of system to also cover issues. Yes, at some point you might want to have a customer support / customer success later to at least triage them, but that makes more sense as you get bigger not when you are small. I actually like having discussions on support days with customer. Yes, sometimes they're more annoying but it's direct feedback of people trying to use your software.
- cdnthrownawy39 2y agoMy on call experience required that I had to be able to respond within 10 ten minutes of the call, with 24/7/365 coverage. But if I couldn't get the issue resolved remotely it meant that I'd have to be in the office lab to recreate and reproduce the problem. It effectively restricted my movements personal movements to stay within commute distance of my office, and that includes all my vacation time as well. That was the better part of a year in my life, continuous. Of constantly considering that every decision, every meal, every movement, every action at all times, and weighing it against the risk of impacting my ability to respond to a customer call. Maintaining an extended period of alertness for a threat that very rarely materializes is frustrating in many ways that I'd like very much to forget. I didn't burn out from it, but it was a major factor in my decision to resign from that company. Obviously people out there that can handle this lifestyle, but I couldn't. And frankly I'm quite content to never try again.
- dmoy 2y agoWas author's big tech experience with Amazon? Because Amazon oncall is ... well it's something. But I'm not sure it's really indicative of the rest of big tech oncall.
- northern-lights 2y agoThe author's Linkedin indeed lists working at AWS for 3 years.
- dmoy 2y agoOk that explains a lot about this article
- rednafi 2y agoA moderate on-call ritual is a necessary evil. I’ve worked at places that tried to get rid of it with all kinds of automation and playbooks, only to revert back to PagerDuty a few months later. That said, my last workplace completely burned me out with a terrible on-call policy and an absurdly short recovery period. Not to mention, upper management tried to gaslight everyone into thinking on-call was just part of normal work and didn’t warrant additional compensation. I get why it’s necessary, but I’m lucky enough to be in a position where I can flat-out decline any job that mandates unpaid on-call.
- lolinder 2y ago> upper management tried to gaslight everyone into thinking on-call was just part of normal work and didn’t warrant additional compensation. If it's made explicitly clear in the job interview process that on call is expected in a role and if it's enforced evenly across the organization, this can absolutely be normal and ethical—you're on a specific salary and your salary includes on call expectations. If you knew about that going in then you weighed it against the offered salary and decided it was worth it. It's not "unpaid on-call" in that case, it's paid on-call. Obviously it can also be done poorly in ways that are harmful and dishonest. Not communicating it clearly during the job interview and salary negotiation, applying it inconsistently, or changing the frequency or the difficulty of the rotation after you have started are all real problems.
- rednafi 2y agoI agree with your expansion. In this particular case, the on-call policy was enforced after the hiring.
- AdieuToLogic 2y ago>> upper management tried to gaslight everyone into thinking on-call was just part of normal work and didn’t warrant additional compensation. > If it's made explicitly clear in the job interview process that on call is expected in a role and if it's enforced evenly across the organization, this can absolutely be normal and ethical—you're on a specific salary and your salary includes on call expectations. Where this logic breaks down is when on-call expectations are in addition to what a salaried position compensates - a 40-ish hour work week. Expecting an employee to be paid industry rate for services rendered and then expecting periodic 24-168 hour near real-time availability without compensation is, by definition, "unpaid on-call".
- snapetom 2y agoI'll add to this some companies (my current one) still has a hero culture. People that are forced to work all weekend to fix a problem are lauded as the ones who went the extra mile. An ego boost and kudos, for some reason are reward enough. Companies don't have substantially reward these employees nor does the company have to invest in paying technical debt, infrastructure, or automation/monitoring. It's incentivisation gone horribly wrong.
- smcameron 2y agoBeen a few years since I worked at Google as an SRE, but I did not find "There’s no incentive in big tech to write software with no bugs" to be particularly true. Perhaps because I was an SRE for a few of the of the older, intrinsic, core products. A lot of the pages we got were for things outside our control (e.g. some fiber optic cables got broken (we found out later), so we've got to drain some cluster because nothing's getting through there, or something's getting overloaded because it's the superbowl today, so throw more machines/memory at it, or other weird external things.) I don't remember a lot of pages due to outright bugs in the code... though it's been enough years now that I might have forgotten.
- nijave 2y agoWere the systems designed to scale based on load and handle transient failures? It seems like lack of automated remediation would be a bug unless it's an "accepted business risk" i.e. cheaper to throw people at to manually fix than build a software solution.
- Xylakant 2y agoYou can’t plan for all failure modes. Weird shit happens and it needs human intervention to figure out what went wrong. Sometimes someone needs to assess what path forward is the lowest (financial) harm and weigh the options. No computer should make that call.
- VirusNewbie 2y agoIf a bug is causing multiple pages, SRE will absolutely either force the devs to fix it, or fix the bug themselves.
- AdieuToLogic 2y agoThe industry myth of "devs need to be on-call just in case prod crashes at 3am" needs to die. First, a system failure addressed by someone awoken "at 3am" assumes said person can transition from sleep to peek analytic mode in moments. This is obviously not the case. Second, a system failure addressed by someone awoken "at 3am" assumes said person possesses omniscient awareness of all aspects of a non-trivial system. This is obviously not the case. Third, and finally, a system failure addressed by someone "at 3am" assumes said person can rectify the problem in real time and without any coworker collaboration nor stakeholder input. The last is most important: If a resolution requires a system change, what organization would allow one person at 3am to make it and push to prod?
- hn_user82179 2y agogreat points. I think over the 8 years of my SRE experience, I've probably caused a few outages after being paged at 3-4am and/or prolonged them. I've fallen asleep during one responding time (fortunately the main outage had been fixed but we went way beyond our internal SLO for backups as I had fallen asleep before running them). That said, the author neglected to mention timezones or following the sun. A 12/12 hour shift or 8/8/8 (NA/EU, NA/EU/EMEA) addresses the sleep deprivation problem pretty well, and is pretty easy to staff in a large enough org
- AdieuToLogic 2y ago> That said, the author neglected to mention timezones or following the sun. A 12/12 hour shift or 8/8/8 (NA/EU, NA/EU/EMEA) addresses the sleep deprivation problem pretty well I completely agree. The "get a call at 3am" scenario, to me, is shorthand for an organization which intentionally under-staffs in order to save money. If a system has genuine 24x7 support needs, be it SLA's or inferior construction, it is incumbent upon the organization to staff accordingly. Still and all, identifying a production issue is one thing, expecting near real-time resolution by personnel likely unfamiliar with the intricacies of every aspect of a non-trivial production system is another.
- 2y ago
- chilldsgn 2y ago> Filing for a new feature implementation would require a thorough documentation, rightfully so, followed by a political campaign to convince the political party of principal engineers and managers to accept the new feature. These stakeholders carry incentives and principals of their own - that do not necessarily always align with the true engineering spirit of solving the problem, nor with satisfying the customer. > The same friction applied to fixing a bug or a flawed process. I would reproduce the bug: spin up the entire environment, the appropriate binary artifact and the reproduced state of application, create the test cases and pin-point the exact problem for the stakeholders as well as present the possible solutions. I would get sent to a dozen of meetings, bouncing my ideas back and forth until receiving the dire verdict - rejection to fix this bug altogether. This has been my experience for the entire duration of my tenure at the current place I am employed at. I've stopped doing this, because my backlog is filled with "best effort" features and when I attempt to slot some of these into a quiet sprint, management says no. Many features I request is not from assumption, but data I gathered with analytics for the customer-facing website. One example is to change a page layout and add better search functionality for this page and its data, because I noticed a +70% no-results rate in the search analytics. I suggested a change but marketing shot it down saying it's low priority for them, while they frequently say that the website can perform better at generating leads. I might be wrong, but to me it feels like utter short-sightedness and goes against the strategic goal of the company. I'm just an engineer, what do I know?
- aorloff 2y agoFirst rule of on-call : if your team has it, then you either take shifts during work hours or are being paid a metric shitload.
- VirusNewbie 2y agoAt google, SRE pays quite well and you don’t have more than 12 hour shifts.
- EZ-E 2y agoOr your job pays extra for on-call shifts outside work hours
- brikym 2y agoI work for a big tech and your solution has already been built. The on-call software uses an LLM to summarize the tickets, and a vector db to find troubleshooting guides and similar tickets.
- jldugger 2y agoNo offense, but this is wildly underselling the goals of oncall SRE. LLMs are extremely crappy about causal analysis, or even just mitigation techniques for services that haven't been widely discussed on stackoverflow.com (i.e. your service). > Creating an on-call process to manually inspect errors in test suites is more valuable than improving the project to be more reliable, as you can directly measure the amount of tests that failed on a weekly basis. It is measurable and presentable to the upper management. You can also measure requests that failed on a weekly basis, and I do. In fact, I added a dashboard panel to do exactly that today for a service (10 years old!!) on a new team I just reorg'd into. I did this because I was annoyed to discover the first (internal) customer outage report of the day could have been repaired by the east coast half of the team team hours before the west coast QA team logged in for the day, but they were unaware anything was wrong. This is a trivial promQL query to implement, and yet it wasn't until today. The problem isn't visibility but risk -- what if you make reliability fixes but the data gets worse? This is not hypothetical, a Youtube engineer documented a similar tale[1]. You can also imagine all kinds of fixes that sound good on paper but can produce paradoxical outcomes (i.e. adding retries causes a metastable failure state[2]). And heck, what if you make no changes, and the numbers decline all on their own? Are you going to scuttle this quarters project work (and promotion fodder!) just to bring this KPI back to normal? Of course, all numbers, even the test suite pass rate, come with risks of missing targets, so the incentives are to commit to reporting as few of them as possible. > tools automate the mundane tasks of an on-call engineer: searching for issues related to a customer report, tracking related software (or hardware) crashes, verifying if the current issue that arose during an on-call is a regression or a known bug and so on. I have a coworker trying to use LLMs for ticket triage, but there's a huge GIGO risk here. Very few people correctly fill in ticket metadata, and even among the more diligent set there will be disagreement. Try an experiment: pick 10 random tickets, and route copies to two of your most diligent ticket workers. Then see how closely their metadata agrees. Is it P1 or P3? Is the bug reported against the puppet repo or the LB repo? Is a config change feature work, bug fix, or testing? Do they dupe known issues, and if to, to the same ticket, or do they just close it as a NTBF known issue? If these two can't agree on basics, then your fine tuning is essentially just additional entropy. Worse, you can't even really measure quality without this messy dataset, and the correct answers should change over time as the software and network architecture evolves. [1]: https://blog.chriszacharias.com/page-weight-matters https://blog.chriszacharias.com/page-weight-matters [2]: https://www.usenix.org/publications/loginonline/metastable-failures-wild https://www.usenix.org/publications/loginonline/metastable-f...
- rohgpt 2y agoI used to be on-call too. I still have muscle memory of the buzz that’d make me leap off my couch like my house was on fire. Except it wasn’t. Just prod, doing prod things. We all know it. You trade your peace of mind, sleep, and sometimes personal freedom (hello, “don’t go beyond cell signal” life). And I agree with many of you—it shouldn’t be this way. We’ve seen teams turn their on-call rotations into something way more humane. Not perfect (nothing in ops is, let’s be real), but sustainable. At Zenduty, we work with a lot of SREs, DevOps folks, and incident commanders who’ve said, “We can’t keep doing this.” So they started rethinking on-call. One of our partners cut their alert volume by 50%. Half! That’s not a magic trick. Just better processes, better tooling, and some good old-fashioned empathy for the humans behind the screens. And if your team’s not giving you breathing room after an all-nighter? You deserve better. I’ll just say it. I saw some of you mention how this stuff becomes political or gets deprioritized in favor of shiny new features. Ugh. I feel that in my bones. The best teams we’ve worked with treat incidents as opportunities to make things better. Every alert is basically your system saying “Help me, I’m tired.” And when you listen and fix it? Less 3 AM wake-ups. Win-win. Anyway, I didn’t mean to ramble, but this topic is close to home. If you’re still stuck in on-call hell, or just want to vent about it. Always happy to swap war stories or chat about how folks are making on-call suck less. Stay sane out there, Rohan DevRel @ Zenduty
- chgs 2y agoWhy don’t you have shift work?
- dakiol 2y agoFor me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is further than a 10 min. trip). Basically, all I can do is stay at home and be available. It sucks, and the money is not worth it.
- rwmj 2y agoThis is why people used to be paid time-and-a-half or even double-time for being on call. Ask your union to demand that. https://en.wikipedia.org/wiki/Time-and-a-half https://en.wikipedia.org/wiki/Time-and-a-half
- chgs 2y agoLast time I was on call it was a one hour payment per day, a 4 hour payment for answering the phone. That lasted about 3 months, just not acceptable.
- liveoneggs 2y agoFor nurses on-call is a tiny amount - $3/hr and then you get 1.5x if you actually get called in
- darrin 2y agoWhen I was at a small IT consulting shop ~15 years ago, this is roughly how it worked. We'd get paid 24x7 for a week on-call at minimum wage + 1.5x normal wage for any hours we had to log in.
- rwmj 2y agoPay is all about power. Nurses individually have little power (as they are replaceable), which is why unionising is good for them as the union gives them collective power. Software engineers are an interesting case; some have a great deal of domain-specific knowledge, giving them leverage over their employers. Many less so, and so a union could help. AI might change this equation too.
- xyst 2y agoQuestion to tech folks in general: when you are "on-call" are you provided with differentials in pay regardless of whether an incident actually occurs? I have seen it vary across companies: - Some companies don’t pay any differential - Some companies only pay out _per incident_ - One company I worked with paid out per shift of on-call rotation - One company treated it as overtime pay
- inetknght 2y ago> Some companies don’t pay any differential This is quite scummy unless your role's primary duties are support and nothing else. I would count Site Reliability Engineer as support being a primary duty, while I would not count Software Engineer as support being a primary duty. > Some companies only pay out _per incident_ That's a bad incentive. It incentivices more incidents to occur. > One company I worked with paid out per shift of on-call rotation Sounds decent on paper but everyone has their breaking point! > One company treated it as overtime pay Sounds decent on paper but everyone has their breaking point!
- zombot 2y agoI must have missed the "breaking up" part. Anyway, the article is just a really long ad for the author.
- eweise 2y agoI'm pretty confident the idea of on-call would go away if the company had to pay for your time. But being salaried, they assume you work for them 24/7.
- inetknght 2y agoYup, that was my experience as described in another comment: https://news.ycombinator.com/item?id=43401441 https://news.ycombinator.com/item?id=43401441
- AStonesThrow 2y agoI was at the tender age of 21, and a mere contractor at my first job. I recall being paid hourly, but never filling a time card. I had just been promoted to a systems admin role, after demonstrating prowess with coding and system configuration, and also the actual qualified admin had quit. One of my projects involved a rewrite of server code for efficiency as our /etc/passwd balooned to thousands of users, and too many logging in simultaneously. So on Thanksgiving Day, we just got in the door with Mom after shopping, and my supervisor rings the home phone; his first words are "There's a hacker on [production shell server]". And I told him a few commands to try, and we did not seem to suffer downtime or need our backups. That may have been the most consequential job I had for a decade. There were thousands of customers depending on us giving "four nines" of available service, including the corporate customers, but we had no formal on-call agreement, not even a way for me to access systems from outside the office, and I was still a novice after a few college courses and a lot of unstructured hacking. My [salaried] father, on the other hand, often brought a pager [which never seemed to beep] home, and one day we were all sitting around the living room when an explosion rattled our windows. We were still a bit rattled ourselves when the phone rang, and it was his cue to go down to the site and respond as a health & safety officer. So my mother said that our household policy would henceforth be: "if something explodes out there, nobody answers the phone!"
- inetknght 2y ago> So my mother said that our household policy would henceforth be: "if something explodes out there, nobody answers the phone!" So if you had family that needed your immediate help, they would not receive it? That's indeed a perverse incentive.
- AStonesThrow 2y agoI'm unsure how this got flipped turned upside-down, but this was Mom asserting boundaries to Dad's work-life balance. The central controversy of on-call duty is that it leaves the household and family short-staffed, contributing to burnout and "workaholism". We were accustomed to Dad being at work M-F, 9-5, and being available and supporting us at all other times. While a boiler explosion is certainly exceptional circumstances, it can be the "camel's nose in the tent" and eventually, Dad is potentially driving up there every Saturday night that a lab worker spilled 1 gram of mercury. Then he's not eating dinner with us, resting, mowing the lawn, helping Mom to shop or do errands. I'm trying to recall any times Dad responded to a family emergency by leaving work, but it was nearly never necessary, since preparedness and response plans defined his career.
- MisterBastahrd 2y agoHere's my answer to on-call requests: no. And if you ask me to clarify my statement, I will answer: no. Because no is good enough, and it's what I negotiated when I took the job. I don't work for employers who can't manage their staff properly. If you're on-call at a company with any sort of scale, you aren't being managed properly. One of the things I tell hiring managers is that I don't mind working later if we're reaching the end of a project and a late afternoon or two might get the product pushed out of the door sooner, or if I've gotten into the zone and am making reat progress. But these are my choices to make. There will never ever be a time when I will subject myself to being tied to a phone for the purposes of waking up in the middle of the night and messing with servers or software. My time is MY time. I only ever have reciprocal relationships with employers and refuse to upset that balance.
- mertleee 2y agoJust hire an SRE. If you want to piss off your team and incur more costs than the annual salary of one decent SRE... ask your team to do on-call rotations. In this day and age if you're a properly capitalized startup you should have the budget for one decent SRE hire. They often times double as a technical writer and are great at preventing knowledge silos in teams / members of a small team since they have to sort of know everything to be an effective SRE. Best process improvement I ever made for my teams.
- SpicyLemonZest 2y agoI don't follow how a single SRE hire would be able to replace an entire team's oncall rotation.
- 0xbadcafebee 2y agoTypical HN blogspam. A blog post that makes vague but incredibly certain claims, generalizing about an entire industry, with no evidence outside a single person's limited experience. Yet there he is at the end of his post, asking people to hire him for consulting. But it's perfect for HN, because it gives people in the comments something to complain about, driving up engagement. I will admit, though, the SaaS tech industry as a whole has a problem. The cargo cult of tech has convinced itself that SaaS is special. That it's never done. That it must be never finished, never stable, always changing, always breaking. Like it's some immutable law of nature. This of course is a convenient lie. Before every business was an online service, software worked like every other product in the world. You built it to do a thing, you tested it thoroughly to ensure it didn't have defects, and then you coordinated one large release. No constant changes or late Friday deploys. No schema changes destroying columns. There was no on-call, because there was no 24/7 service. There were floppy disks and CDROMs, and people running software on their own computers, and a very small number of 24/7 enterprise systems shared by lots of users (mostly run by ISPs, and one or two large tech companies and industry bodies). But you can deliver online software like it's desktop software. You can ensure your software is bug-free before you burn your master disk. But then you don't get to skip all those time-consuming tests. Then you don't get to ship your features faster than your competitor. Then you don't get to do A/B testing and micro-tweaks and experiments and feature flags and blue/green deploys. We want to do all the things, at any time, with no consequences. So on-call is still a thing, because people want to have their cake and eat it too. The actual infrastructure on the backend shared by millions of users? Not actually hard to maintain. It's the same shit as 20 years ago, but much, much easier. Make sure the disks don't fill up. Auto-scale the VMs and containers. Design your apps to not exceed network bandwidth. There's really not much else to break. The only thing that breaks a system is changing it, or bugs from not enough testing. So do the testing, and the bare minimum of autoscaling in the cloud, and nothing should ever break in production. But constantly mutating SaaS, without due diligence, is too addictive. Nobody's going to abandon the freedom of doing whatever the fuck they want, just to keep from waking people up in the middle of the night. The business people sure as fuck aren't going to abandon their "competitive advantage". And devs who don't care if their software works or not don't want to abandon untested deploys from their laptops. So on-call is here to stay.
- jyounker 2y agoThe author's description of how on-call works at "big tech" seems to be a complaint about a specific tech company, and the author is hastily generalizing to the rest of the industry. His experience has nothing in common with my on-call experience working at a different "big tech" company.
- a_imho 2y agoIANAL, but my understanding is that in Europe on-call can fall under working time, depending on the exact nature of the requirements. In fact, I'm very interested whether anyone has firsthand experience arguing developer on-call falls under such category. E.g. being 8 minutes away from a work machine and network access is not too dissimilar to a firefighter being 8 minutes away from the fire station. https://www.eurofound.europa.eu/en/european-industrial-relations-dictionary/call-working https://www.eurofound.europa.eu/en/european-industrial-relat...
- staplung 2y agoIt seems that companies often end up with On-Call after they've reached some critical size but decide to do it in the cheapest possible way: let's just make our engineers do it! One big problem is that they tend not to make any attempt to ensure that the on-call rotations are equitable across teams. You might need to be on call one week out of 4 or one week out of 20. At review time, this matters. Since you're typically not getting paid for the on call work or having it factor into your performance review, you're actually incentivized to snooze alerts and kick the can down the road until after your rotation or to join another team with lower rates of on call incidents. Ironically, if you're spending 25% or your time on call and actually being diligent about the role, you're more likely to be adept with the typically hodgepodge collection of tools that you need to use to diagnose and address problems and will be better able to handle incidents than if you only ever use those tools once every 5 months.
- F-W-M 2y agoI am on-call for a week every ~8 weeks. I get 500 € and two additional days of vacation for it. If I do get a call, the additional hours go to my time account. Fair deal, I'd say. Only problem is that most of the applications are from vendors, so getting problems fixed is an ordeal.
- vismit2000 2y agoGetting 404, archived version: https://web.archive.org/web/20250318031523/https://reflector.dev/articles/breaking-up-with-on-call/ https://web.archive.org/web/20250318031523/https://reflector...
- hackburg 2y ago[dead]