5 ms·

> "if you're the only person that 'gives a shit,' YOU'RE GONNA HAVE A BAD TIME." Not only that (which is 100% true) but also you're raising the bar in required

> "if you're the only person that 'gives a shit,' YOU'RE GONNA HAVE A BAD TIME." Not only that (which is 100% true) but also you're raising the bar in required work quality for everyone else to get on par with you, therefore expect to make a few friends and many enemies, especially among colleagues.

ProllyInfamous 3y ago
>expect to make a few friends and many enemies The older I get, the less I'm willing to help disaster scenarios. "A lack of planning on your behalf, does not constitute an emergency upon mine."
- dvt 3y ago
  > "A lack of planning on your behalf, does not constitute an emergency upon mine." Agreed. Imo, having "on call" rotations is a clear red flag and bad software practice that has somehow made it in the mainstream. I understand the idea of "mission critical" when you're running software on the Moon or something, but if your Earthbound software has so many potential bugs that you need to have dedicated people being on call every weekend to fix bugs or restart servers, you just built it poorly.
  - somsak2 3y ago
    If you work at a company on a team with a user-facing surface, you need to have an on-call rotation. Consumer expectations of uptime are extremely high and there's no way to build software that ensures 100% reliability without any human intervention.
    - dvt 3y ago
      Fundamentally disagree with this. In fact, most of human technology has worked without on-call rotations. From the complicated movements of mechanical watches, to the still-ticking heartbeats of the Voyager probes. Most software engineers just happen to be bad engineers. On-call rotations are a band-aid for poor planning and development (to be fair, often imposed by tech-delinquent middle managers or executives).
      - dasil003 3y ago
        Obviously the truth is somewhere in between, there is no engineer so talented that they can produce a 100% reliable and available system, and the percentage goes down the more complex the system gets. The decision of whether to have a on-call rotation should be based on the consequences of downtime, not on some kind of moral stance on human fallibility.
      - Gud 3y ago
        Let’s get rid of on call firemen while we’re at it they shouldn’t really be required. Let’s all plan our emergencies to 8am to 5pm, Monday to Friday. And don’t forget the scheduled lunch break at 1pm!
        lotsofpulp 3y ago
        They can be scheduled on shifts instead of being on call. It just requires spending more money hiring more people.
        _kblcuk_ 3y ago
        Being on call is firemen job, and they do have shifts. In most of the companies you have something akin to construction workers that work from 9 to 5 and then also are expected to be available to do casual firefighting from 5 to 9 because it's "easier".
        dasil003 3y ago
        At a certain scale mature software companies do in fact have dedicated incident managers, and dedicated SREs who work primarily on stability from a more systematic perspective. However they still need support from the application developers due to the nature of software. In the old days operations tended to be very isolated in much the way you are proposing. The problem with this is that stability depends very much on the software, so over time operations folks would be extremely defensive and impose all kinds of constraints on what software could do, and the software engineers would be frustrated that they couldn't do things efficiently. Imagine how firefighters would feel if construction workers had a tendency to randomly leave explosives and gas cans hidden throughout new construction and then waltzed off to the next job while the firefighters had to deal with the consequences. At the end of the day, devs need to have some skin in the game or it's a recipe for disaster.
    - overgard 3y ago
      I only had one experience with this early in my career, but I remember it wasn't good. Basically, we had a 5-6 person team that had to support like 10 important legacy services. One person essentially ended up being on call for a week, having to wake up early to debug issues with a thing you didn't build in the first place and only vaguely understood. Do you think this company had good CI/CD and automated tests? They did not. There was fortunately a lot of monitoring so at least you knew when the service was in a bad state, but absolutely nothing else other than a ticket and an angry customer. I would much rather have extensive test coverage, very good CI/CD, make sure not to do releases on weekends and holidays, and have a few people whose job is to do the monitoring and escalate to the right people rather than just putting a target on a random engineer's back and hope they can fix things quickly.
    - SoftTalker 3y ago
      You think that, but when Github is down, or AWS-East is down, everyone just kind of endures it, waiting until it's back up. And life goes on.
      - alwa 3y ago
        Of course, when a hospital’s IT systems go down, people do indeed have to endure it, but life does not always in fact go on [0]. Horses for courses, I guess. [0] https://www.theverge.com/2021/9/27/22696097/hospital-ransomware-cyberattack-death-rates-patients https://www.theverge.com/2021/9/27/22696097/hospital-ransomw...
        SoftTalker 3y ago
        Hospital IT systems failures are are mostly going to cause problems with scheduling, record-keeping, and billing. By far, the most likely thing to kill you in a hospital is not the IT system but errors by the doctors and nurses. Or that you're too sick to save no matter what they do.
      - somsak2 3y ago
        Right, and the reason "everyone" has confidence in just waiting a bit for it to go back up without panicking and having to try to find an alternative is precisely because GitHub and AWS have on-call engineers that are alerted and immediately work on fixing it.
    - j45 3y ago
      On call rotation is one thing. Everyone being motivated to develop in a way that isn’t resulting in brittle software and breaking and maybe even use boring tech for stability so even if an on call is a real thing, it’s relatively benign. It’s not a bad thing to rely on the genius of smart people who have been at it for decades and are decades ahead in some realizations. Having software that self reports and logs multiple occurrences of the same errors and escalating errors that start in the app are one great way to stay ahead of issues. By the time someone reaches out it’s easy to find the error, session, user, and say ok we see it and are on it. Acknowledgement at this depth upfront quite often let’s the customers to say it’s ok take a look on Monday. It’s reassuring. Also easy to forward such an issue to a distributed team.
  - CityOfThrowaway 3y ago
    Most of the time, this is a useful frame. With that said, there are some types of software that exist to deal with complexity external to itself. External systems (integrations, screen scraping systems, etc.) can break without fault internally, but need to be fixed ASAP. One might say, "You built it poorly by deciding to build this at all." And sometimes that's true. But I've been in plenty of situations where the options were 1) don't solve the customer problem, don't pass go, don't collect $200. Or, 2) build finicky software that works but needs to be babied... and hire engineers that don't mind doing this type of work. That last bit is critical. Some engineers like the heroics and drama of it. Some people can't help but run headfirst into the fire... and there's jobs for them there :-)
    - 8n4vidtmkvmk 3y ago
      I didn't mind staying late some days to see whether or not my migration would blow up production or not. But oncall suuuuuçks. Cleaning up my own mess is one thing, cleaning up some one else's nonsense is another.
      - goldenkey 3y ago
        Oncall done right will involve a feedback mechanism to the originators of the root cause.
    - j45 3y ago
      Things happen all the time. So lending a hand with emergencies that even could have been avoided is ok to help someone new to it. Caring via software or infrastructure for users is often about effort as you have described.
    - tragic 3y ago
      At my last job, there was an incident (thankfully discovered in working hours) caused by a supplier accidentally deleting the country of Austria from some crucial database. You can't anticipate everything like that, but the show must go on ...
  - d_tr 3y ago
    > Imo, having "on call" rotations is a clear red flag and bad software practice that has somehow made it in the mainstream. "Just throw more humans at it!"
  - aksss 3y ago
    I used to run IT operations for a telecom, hundreds of physical servers, facilities, 24x7 operators. Hey, shit happens. Problems with hardware, facilities, software jobs, COTS software, homegrown software. There’s a lot of benefit to having someone on call in each domain of concern rather than letting problems fester or go unresolved all night, particularly if the issues are customer-facing. It’s easy to say “just build your crap to anticipate every failure mode”, but that’s not realistic for large operations, especially if you aren’t sitting on an Apple-size pile of cash and talent.
  - rendall 3y ago
    > Imo, having "on call" rotations is a clear red flag and bad software practice that has somehow made it in the mainstream. I thought so too until I gigged for a SaaS company that absolutely needed it. No, it was not moon-shot rocket control software, but it was a money printer with literally thousands of clients around the globe paying six figures for the service. The on-call SWE duty was not onerous: about 2 hour shifts every 2 months, and your job was to attempt to handle it and then escalate it to staff if you couldn't. There were so many CI tests on the way to production that nothing ever happened, until it did, and then you needed to make sure that money printer go brrr.
  - organsnyder 3y ago
    If your on-call responsibilities are always “fixing bugs or restarting servers”, then indeed you built it poorly. But on-call shifts can also be used for responding to novel situations that couldn’t be planned for in advance. The best on-call rotations are the ones where you’re rarely paged, but when those pages happen they’re for important urgent work where your involvement is vital (even if your role only needs to be channeling the work to the right person). Ideally you’re also compensated for being available, even when nothing happens (my current team gets a day off [in addition to normal PTO] for every week of on-call time).
  - andrewaylett 3y ago
    The expected number of alerts for my team in a normal week on-call (which is out-of-hours only, a different team member covers working hours) is zero. If one of us is paged, there's a good chance it's because of something my team has control over, and we'll try to make sure it doesn't happen a second time. My experience is that a dedicated team running a follow-the-sun rotation (so the team doesn't need much if any "out of hours" because it's always in hours for at least one of them) actually leads to flakier services and more alerts. Without the visibility of the impact of the flaky services, fixes are lower priority and also more difficult to validate. In this manner, on-call is not a lack of planning, nor an emergency, but deliberately ensuring we have a plan in place to triage out-of-hours incidents, mitigate those that need mitigating, and leaving anything else to be fixed in-hours. On the other hand, unpaid on-call is unacceptable.
  - ddalex 3y ago
    This is not a software quality problem, this is a business efficiency problem. We could write commercial software with no bugs, but it would be very expensive (slow process, complicated tooling, etc). It's far cheaper to accept some rate of bugs and pay some schmucks weekends/nights to be OnCall)
- j45 3y ago
  One of my favourite quotes. It can be made more or less direct to great resonance. “If you want my help, plan for it and understand I’m not able to help last minute when there’s no planning”
- kmoser 3y ago
  > "A lack of planning on your behalf, does not constitute an emergency upon mine." Depends on your relationship with the person who planned poorly: if you truly don't care, then you run the risk of becoming known as the jerk engineer who isn't a team player. The reality is that employees are generally expected to cover for each other to provide the (paying) customer with consistent support. If the party who planned poorly was the customer, you have to decide whether they're paying you enough to scramble for them, and whether you want their accolades or their ire. My M.O. is to support my customers through thick and thin (within reason, of course), which tends to get me repeat business and referrals. Yes, going out of your way for your customers is good for business. Give a shit.
- sonicanatidae 3y ago
  Tip: Never get within 200 feet of a Systems Admin position.
j45 3y ago
The “don’t work too hard” crowd when you might be working harder than them and “must be nice” when you are relaxing more than them can both be avoided. People who don’t like anyone working more than them or enjoying life more than them are usually haters who are busy not doing much themselves. Raising the bar doesn’t mean having to make it more work for everyone. Those doubt worshipping and self-sabotaging enemies aren’t worth having in your life but it is nice to give a bit of shit about them too. Move at your pace to wearers what you want and you will find like minded people.

ProllyInfamous 3y ago

>expect to make a few friends and many enemies The older I get, the less I'm willing to help disaster scenarios. "A lack of planning on your behalf, does not constitute an emergency upon mine."

dvt 3y ago
> "A lack of planning on your behalf, does not constitute an emergency upon mine." Agreed. Imo, having "on call" rotations is a clear red flag and bad software practice that has somehow made it in the mainstream. I understand the idea of "mission critical" when you're running software on the Moon or something, but if your Earthbound software has so many potential bugs that you need to have dedicated people being on call every weekend to fix bugs or restart servers, you just built it poorly.
- somsak2 3y ago
  If you work at a company on a team with a user-facing surface, you need to have an on-call rotation. Consumer expectations of uptime are extremely high and there's no way to build software that ensures 100% reliability without any human intervention.
  - dvt 3y ago
    Fundamentally disagree with this. In fact, most of human technology has worked without on-call rotations. From the complicated movements of mechanical watches, to the still-ticking heartbeats of the Voyager probes. Most software engineers just happen to be bad engineers. On-call rotations are a band-aid for poor planning and development (to be fair, often imposed by tech-delinquent middle managers or executives).
    - dasil003 3y ago
      Obviously the truth is somewhere in between, there is no engineer so talented that they can produce a 100% reliable and available system, and the percentage goes down the more complex the system gets. The decision of whether to have a on-call rotation should be based on the consequences of downtime, not on some kind of moral stance on human fallibility.
    - Gud 3y ago
      Let’s get rid of on call firemen while we’re at it they shouldn’t really be required. Let’s all plan our emergencies to 8am to 5pm, Monday to Friday. And don’t forget the scheduled lunch break at 1pm!
      - lotsofpulp 3y ago
        They can be scheduled on shifts instead of being on call. It just requires spending more money hiring more people.
      - _kblcuk_ 3y ago
        Being on call is firemen job, and they do have shifts. In most of the companies you have something akin to construction workers that work from 9 to 5 and then also are expected to be available to do casual firefighting from 5 to 9 because it's "easier".
        dasil003 3y ago
        At a certain scale mature software companies do in fact have dedicated incident managers, and dedicated SREs who work primarily on stability from a more systematic perspective. However they still need support from the application developers due to the nature of software. In the old days operations tended to be very isolated in much the way you are proposing. The problem with this is that stability depends very much on the software, so over time operations folks would be extremely defensive and impose all kinds of constraints on what software could do, and the software engineers would be frustrated that they couldn't do things efficiently. Imagine how firefighters would feel if construction workers had a tendency to randomly leave explosives and gas cans hidden throughout new construction and then waltzed off to the next job while the firefighters had to deal with the consequences. At the end of the day, devs need to have some skin in the game or it's a recipe for disaster.
  - overgard 3y ago
    I only had one experience with this early in my career, but I remember it wasn't good. Basically, we had a 5-6 person team that had to support like 10 important legacy services. One person essentially ended up being on call for a week, having to wake up early to debug issues with a thing you didn't build in the first place and only vaguely understood. Do you think this company had good CI/CD and automated tests? They did not. There was fortunately a lot of monitoring so at least you knew when the service was in a bad state, but absolutely nothing else other than a ticket and an angry customer. I would much rather have extensive test coverage, very good CI/CD, make sure not to do releases on weekends and holidays, and have a few people whose job is to do the monitoring and escalate to the right people rather than just putting a target on a random engineer's back and hope they can fix things quickly.
  - SoftTalker 3y ago
    You think that, but when Github is down, or AWS-East is down, everyone just kind of endures it, waiting until it's back up. And life goes on.
    - alwa 3y ago
      Of course, when a hospital’s IT systems go down, people do indeed have to endure it, but life does not always in fact go on [0]. Horses for courses, I guess. [0] https://www.theverge.com/2021/9/27/22696097/hospital-ransomware-cyberattack-death-rates-patients https://www.theverge.com/2021/9/27/22696097/hospital-ransomw...
      - SoftTalker 3y ago
        Hospital IT systems failures are are mostly going to cause problems with scheduling, record-keeping, and billing. By far, the most likely thing to kill you in a hospital is not the IT system but errors by the doctors and nurses. Or that you're too sick to save no matter what they do.
    - somsak2 3y ago
      Right, and the reason "everyone" has confidence in just waiting a bit for it to go back up without panicking and having to try to find an alternative is precisely because GitHub and AWS have on-call engineers that are alerted and immediately work on fixing it.
  - j45 3y ago
    On call rotation is one thing. Everyone being motivated to develop in a way that isn’t resulting in brittle software and breaking and maybe even use boring tech for stability so even if an on call is a real thing, it’s relatively benign. It’s not a bad thing to rely on the genius of smart people who have been at it for decades and are decades ahead in some realizations. Having software that self reports and logs multiple occurrences of the same errors and escalating errors that start in the app are one great way to stay ahead of issues. By the time someone reaches out it’s easy to find the error, session, user, and say ok we see it and are on it. Acknowledgement at this depth upfront quite often let’s the customers to say it’s ok take a look on Monday. It’s reassuring. Also easy to forward such an issue to a distributed team.
- CityOfThrowaway 3y ago
  Most of the time, this is a useful frame. With that said, there are some types of software that exist to deal with complexity external to itself. External systems (integrations, screen scraping systems, etc.) can break without fault internally, but need to be fixed ASAP. One might say, "You built it poorly by deciding to build this at all." And sometimes that's true. But I've been in plenty of situations where the options were 1) don't solve the customer problem, don't pass go, don't collect $200. Or, 2) build finicky software that works but needs to be babied... and hire engineers that don't mind doing this type of work. That last bit is critical. Some engineers like the heroics and drama of it. Some people can't help but run headfirst into the fire... and there's jobs for them there :-)
  - 8n4vidtmkvmk 3y ago
    I didn't mind staying late some days to see whether or not my migration would blow up production or not. But oncall suuuuuçks. Cleaning up my own mess is one thing, cleaning up some one else's nonsense is another.
    - goldenkey 3y ago
      Oncall done right will involve a feedback mechanism to the originators of the root cause.
  - j45 3y ago
    Things happen all the time. So lending a hand with emergencies that even could have been avoided is ok to help someone new to it. Caring via software or infrastructure for users is often about effort as you have described.
  - tragic 3y ago
    At my last job, there was an incident (thankfully discovered in working hours) caused by a supplier accidentally deleting the country of Austria from some crucial database. You can't anticipate everything like that, but the show must go on ...
- d_tr 3y ago
  > Imo, having "on call" rotations is a clear red flag and bad software practice that has somehow made it in the mainstream. "Just throw more humans at it!"
- aksss 3y ago
  I used to run IT operations for a telecom, hundreds of physical servers, facilities, 24x7 operators. Hey, shit happens. Problems with hardware, facilities, software jobs, COTS software, homegrown software. There’s a lot of benefit to having someone on call in each domain of concern rather than letting problems fester or go unresolved all night, particularly if the issues are customer-facing. It’s easy to say “just build your crap to anticipate every failure mode”, but that’s not realistic for large operations, especially if you aren’t sitting on an Apple-size pile of cash and talent.
- rendall 3y ago
  > Imo, having "on call" rotations is a clear red flag and bad software practice that has somehow made it in the mainstream. I thought so too until I gigged for a SaaS company that absolutely needed it. No, it was not moon-shot rocket control software, but it was a money printer with literally thousands of clients around the globe paying six figures for the service. The on-call SWE duty was not onerous: about 2 hour shifts every 2 months, and your job was to attempt to handle it and then escalate it to staff if you couldn't. There were so many CI tests on the way to production that nothing ever happened, until it did, and then you needed to make sure that money printer go brrr.
- organsnyder 3y ago
  If your on-call responsibilities are always “fixing bugs or restarting servers”, then indeed you built it poorly. But on-call shifts can also be used for responding to novel situations that couldn’t be planned for in advance. The best on-call rotations are the ones where you’re rarely paged, but when those pages happen they’re for important urgent work where your involvement is vital (even if your role only needs to be channeling the work to the right person). Ideally you’re also compensated for being available, even when nothing happens (my current team gets a day off [in addition to normal PTO] for every week of on-call time).
- andrewaylett 3y ago
  The expected number of alerts for my team in a normal week on-call (which is out-of-hours only, a different team member covers working hours) is zero. If one of us is paged, there's a good chance it's because of something my team has control over, and we'll try to make sure it doesn't happen a second time. My experience is that a dedicated team running a follow-the-sun rotation (so the team doesn't need much if any "out of hours" because it's always in hours for at least one of them) actually leads to flakier services and more alerts. Without the visibility of the impact of the flaky services, fixes are lower priority and also more difficult to validate. In this manner, on-call is not a lack of planning, nor an emergency, but deliberately ensuring we have a plan in place to triage out-of-hours incidents, mitigate those that need mitigating, and leaving anything else to be fixed in-hours. On the other hand, unpaid on-call is unacceptable.
- ddalex 3y ago
  This is not a software quality problem, this is a business efficiency problem. We could write commercial software with no bugs, but it would be very expensive (slow process, complicated tooling, etc). It's far cheaper to accept some rate of bugs and pay some schmucks weekends/nights to be OnCall)
j45 3y ago
One of my favourite quotes. It can be made more or less direct to great resonance. “If you want my help, plan for it and understand I’m not able to help last minute when there’s no planning”
kmoser 3y ago
> "A lack of planning on your behalf, does not constitute an emergency upon mine." Depends on your relationship with the person who planned poorly: if you truly don't care, then you run the risk of becoming known as the jerk engineer who isn't a team player. The reality is that employees are generally expected to cover for each other to provide the (paying) customer with consistent support. If the party who planned poorly was the customer, you have to decide whether they're paying you enough to scramble for them, and whether you want their accolades or their ire. My M.O. is to support my customers through thick and thin (within reason, of course), which tends to get me repeat business and referrals. Yes, going out of your way for your customers is good for business. Give a shit.
sonicanatidae 3y ago
Tip: Never get within 200 feet of a Systems Admin position.

j45 3y ago

The “don’t work too hard” crowd when you might be working harder than them and “must be nice” when you are relaxing more than them can both be avoided. People who don’t like anyone working more than them or enjoying life more than them are usually haters who are busy not doing much themselves. Raising the bar doesn’t mean having to make it more work for everyone. Those doubt worshipping and self-sabotaging enemies aren’t worth having in your life but it is nice to give a bit of shit about them too. Move at your pace to wearers what you want and you will find like minded people.