6 ms·

I would go so far as to say it is a best practice. It aligns incentives. If you know you're going to have to wake up at 2am when your code blows up, you are mor

by dyeje 5y ago

I would go so far as to say it is a best practice. It aligns incentives. If you know you're going to have to wake up at 2am when your code blows up, you are more likely to make it resilient.

manuelabeledo 5y ago
> If you know you're going to have to wake up at 2am when your code blows up, you are more likely to make it resilient. Isn't it also true that a developer who routinely misses sleep, is less productive and makes more mistakes during working hours?
- dyeje 5y ago
  On call incidents shouldn't be routine. A rotation spreads responsibility across the team. Ultimately, someone has to be responsible for incidents and it aligns incentives best for it to be the engineers who maintain the code.
  - manuelabeledo 5y ago
    > it aligns incentives best for it to be the engineers who maintain the code. How many at a time? It has to be two or more, otherwise anyone could push changes to production at any time, without supervision.
    - dyeje 5y ago
      I recommend one per team. You want somebody who is knowledgeable on the specifics of any part of the system that could break. I don't think having more than 1 is a requirement though, if we're talking about small companies. As long as there is a "break glass in case of emergency" way to push to production that is auditable.
      - manuelabeledo 5y ago
        Do you trust every single dev to push the right workaround?
- yuppie_scum 5y ago
  Why is your productivity more valuable than the Ops/SRE team that probably costs the company more to employ?
  - manuelabeledo 5y ago
    It may not be. It all depends on how much value you put on resilience and quality in software. Again, a sleep deprived developer has a higher chance of making mistakes, than a rested one. But I have heard your argument before, and it kind of rings hollow. Unless you are willing to allow developers to push changes to production in the middle of the night and without supervision, what exactly is the benefit of having them on-call? Saving money?
    - yuppie_scum 5y ago
      What exactly is the benefit of having a non developer on call who has no fucking clue what the problem is because they didn’t write the code, they don’t know the AC’s, etc? That doesn’t save any money or productivity either. Your point is correct thought that the responder does need the ability to make a hot fix. Non actionable alerts are of negative value to the company. You should read The Phoenix Project, that will help you understand the rationale.
      - manuelabeledo 5y ago
        > You should read The Phoenix Project, that will help you understand the rationale. What makes you think I haven't read it? When my company went through its so called "devops transformation", first thing our devops advocate did was giving away a dozen or so copies. I understand the rationale. I just happen to not agree with it. In my mind, it's naive to believe that a team would have more than two or three competent coders, with a thorough knowledge of the code base, the high level vision of where that code fits and interacts with the rest of the infrastructure, the skills to quickly fix things, and, on top of that, the willing to be on-call several days a month. And then there is the fact that nobody looks for solutions at 2am, just workarounds.
lostdog 5y ago
Then the CEO should also be woken up to be sure the incentives are truly aligned. After all, it is their website that is blowing up, and being woken at 2am will make them more likely to ensure best practices company wide.
- dyeje 5y ago
  If the CEO is equipped to diagnose and resolve incidents, then yes absolutely they should be in the on call rotation.
actionherobiju 5y ago
I agree with this, however companies should pay extra for the on-call hours. In our team, I am on-call once every 6 weeks. So I am "working" 9 weekends a year. As long as I am paid for 18 days, which is almost 3/4 of a month (considering 22 working days a month) ,I am cool with it. But most companies dont.

by dyeje 5y ago

I would go so far as to say it is a best practice. It aligns incentives. If you know you're going to have to wake up at 2am when your code blows up, you are more likely to make it resilient.

manuelabeledo 5y ago

> If you know you're going to have to wake up at 2am when your code blows up, you are more likely to make it resilient. Isn't it also true that a developer who routinely misses sleep, is less productive and makes more mistakes during working hours?

dyeje 5y ago
On call incidents shouldn't be routine. A rotation spreads responsibility across the team. Ultimately, someone has to be responsible for incidents and it aligns incentives best for it to be the engineers who maintain the code.
- manuelabeledo 5y ago
  > it aligns incentives best for it to be the engineers who maintain the code. How many at a time? It has to be two or more, otherwise anyone could push changes to production at any time, without supervision.
  - dyeje 5y ago
    I recommend one per team. You want somebody who is knowledgeable on the specifics of any part of the system that could break. I don't think having more than 1 is a requirement though, if we're talking about small companies. As long as there is a "break glass in case of emergency" way to push to production that is auditable.
    - manuelabeledo 5y ago
      Do you trust every single dev to push the right workaround?
yuppie_scum 5y ago
Why is your productivity more valuable than the Ops/SRE team that probably costs the company more to employ?
- manuelabeledo 5y ago
  It may not be. It all depends on how much value you put on resilience and quality in software. Again, a sleep deprived developer has a higher chance of making mistakes, than a rested one. But I have heard your argument before, and it kind of rings hollow. Unless you are willing to allow developers to push changes to production in the middle of the night and without supervision, what exactly is the benefit of having them on-call? Saving money?
  - yuppie_scum 5y ago
    What exactly is the benefit of having a non developer on call who has no fucking clue what the problem is because they didn’t write the code, they don’t know the AC’s, etc? That doesn’t save any money or productivity either. Your point is correct thought that the responder does need the ability to make a hot fix. Non actionable alerts are of negative value to the company. You should read The Phoenix Project, that will help you understand the rationale.
    - manuelabeledo 5y ago
      > You should read The Phoenix Project, that will help you understand the rationale. What makes you think I haven't read it? When my company went through its so called "devops transformation", first thing our devops advocate did was giving away a dozen or so copies. I understand the rationale. I just happen to not agree with it. In my mind, it's naive to believe that a team would have more than two or three competent coders, with a thorough knowledge of the code base, the high level vision of where that code fits and interacts with the rest of the infrastructure, the skills to quickly fix things, and, on top of that, the willing to be on-call several days a month. And then there is the fact that nobody looks for solutions at 2am, just workarounds.

lostdog 5y ago

Then the CEO should also be woken up to be sure the incentives are truly aligned. After all, it is their website that is blowing up, and being woken at 2am will make them more likely to ensure best practices company wide.

dyeje 5y ago
If the CEO is equipped to diagnose and resolve incidents, then yes absolutely they should be in the on call rotation.

actionherobiju 5y ago

I agree with this, however companies should pay extra for the on-call hours. In our team, I am on-call once every 6 weeks. So I am "working" 9 weekends a year. As long as I am paid for 18 days, which is almost 3/4 of a month (considering 22 working days a month) ,I am cool with it. But most companies dont.