7 ms·

Yea pretty much from my experience now in the US, most private companies use their salaried employees and wake them up multiple times a night as sometimes with

Yea pretty much from my experience now in the US, most private companies use their salaried employees and wake them up multiple times a night as sometimes with customer directly on the other end of the line. Pagerduty enables this to work. This is considered now normal for smallish private orgs, shit my spot just got bought by a billions dollar defense contractor I still get woken up at 3am with a end user having printer issues.. and I work as a devops engineer.

softawre 8y ago
If you're not exaggerating you need to nip that in the bud. Printer issues at 3am? Have two classes of alerts and send printer issues to somebody else, or worst case email notification for the morning.
- sambull 8y ago
  Sev1 for us, end users are 24/7 manufacturers and they consider this a business stoppage, this would be SLA downtime. They shoved old school COTs into AWS and called it cloud. Which ugh yea.
wil421 8y ago
Do your incidents have Priorites (Severity in the article)? At my company a printer would never be a P1 to P2 therefore no one gets paged at 3am due to printer issues. It’s not full proof and my team did get a P1 due to a field length being too short but it was during normal business hours. It was quickly changed to a P4 and the person who opened it got a lesson on Incident priorities.
- sambull 8y ago
  Printing would be Sev 1, as our end users are 24/7 JIT manufacturers with big contracts. Without printers, no shipping labels, packaging, or inventory barcodes etc for their normal manufacturing shifts.
  - wil421 8y ago
    I understand now. The company I used to work for was in manufacturing. Luckily my current employer is all services and software.
  - flukus 8y ago
    Presumably there are multiple redundant printers and sev 1 would only be an outage effecting all of them?
  - C1sc0cat 8y ago
    So you need a 24/7 ops coverage to handle that and not rely on calling out an expensive M&P grade.
AmericanChopper 8y ago
As long as it’s managed properly, I don’t see how this is a problem in general. Small to medium sized companies simply don’t have the resources to have a full time 24 hour operations center. That’s at least two shifts, with at least two people each, so 4 salaried engineers just to cover after hours ops. If you have 10 engineers in your company, you’re not going to put 40% of your engineering resources into after hours operations, and that doesn’t even account for when a subject matter expert is required to help respond. Where I work, there’s an on call roster, that people volunteer to be on, and you’re paid extra for being there. We also have a good post-mortem culture, so after hours pages also track down steadily over time. Unless the company happens to be geographically distributed just right to avoid needing this, I really don’t see the problem with these kinds of arrangements.
- marcosdumay 8y ago
  You need one extra person to cover the night hours, and you need to alternate weekends between your staff. Of course, you can't have only one person on the position, you need at least two (but then, you would need people testing your operations anyway). Optimally, you will alternate them between night and day hours so they integrate with the rest of the team. I don't understand how US companies are allowed to not do that.
  - donavanm 8y ago
    Have you tried that before? Theres the obvious, that “night” is longer than business hours. Then theres the squishy people problems of things like sick time, meal breaks, using the WC, holiday, or PTO. And ignoring all that what do you do when there are two discrete problems at the same time? Varying shift times is terrible for humans. In short, no 1 headcount does not 24/7 coverage make. I dont even believe the grandparents 2 per rota is enough to be sustainable. I have seen as small as 4 work acceptably.
    - quicksilver03 8y ago
      A million times this. I'm about to tackle this issue at my current gig, and I won't have any on-call rotation with less than 4 people.

softawre 8y ago

If you're not exaggerating you need to nip that in the bud. Printer issues at 3am? Have two classes of alerts and send printer issues to somebody else, or worst case email notification for the morning.

sambull 8y ago
Sev1 for us, end users are 24/7 manufacturers and they consider this a business stoppage, this would be SLA downtime. They shoved old school COTs into AWS and called it cloud. Which ugh yea.

wil421 8y ago

Do your incidents have Priorites (Severity in the article)? At my company a printer would never be a P1 to P2 therefore no one gets paged at 3am due to printer issues. It’s not full proof and my team did get a P1 due to a field length being too short but it was during normal business hours. It was quickly changed to a P4 and the person who opened it got a lesson on Incident priorities.

sambull 8y ago
Printing would be Sev 1, as our end users are 24/7 JIT manufacturers with big contracts. Without printers, no shipping labels, packaging, or inventory barcodes etc for their normal manufacturing shifts.
- wil421 8y ago
  I understand now. The company I used to work for was in manufacturing. Luckily my current employer is all services and software.
- flukus 8y ago
  Presumably there are multiple redundant printers and sev 1 would only be an outage effecting all of them?
- C1sc0cat 8y ago
  So you need a 24/7 ops coverage to handle that and not rely on calling out an expensive M&P grade.

AmericanChopper 8y ago

As long as it’s managed properly, I don’t see how this is a problem in general. Small to medium sized companies simply don’t have the resources to have a full time 24 hour operations center. That’s at least two shifts, with at least two people each, so 4 salaried engineers just to cover after hours ops. If you have 10 engineers in your company, you’re not going to put 40% of your engineering resources into after hours operations, and that doesn’t even account for when a subject matter expert is required to help respond. Where I work, there’s an on call roster, that people volunteer to be on, and you’re paid extra for being there. We also have a good post-mortem culture, so after hours pages also track down steadily over time. Unless the company happens to be geographically distributed just right to avoid needing this, I really don’t see the problem with these kinds of arrangements.

marcosdumay 8y ago
You need one extra person to cover the night hours, and you need to alternate weekends between your staff. Of course, you can't have only one person on the position, you need at least two (but then, you would need people testing your operations anyway). Optimally, you will alternate them between night and day hours so they integrate with the rest of the team. I don't understand how US companies are allowed to not do that.
- donavanm 8y ago
  Have you tried that before? Theres the obvious, that “night” is longer than business hours. Then theres the squishy people problems of things like sick time, meal breaks, using the WC, holiday, or PTO. And ignoring all that what do you do when there are two discrete problems at the same time? Varying shift times is terrible for humans. In short, no 1 headcount does not 24/7 coverage make. I dont even believe the grandparents 2 per rota is enough to be sustainable. I have seen as small as 4 work acceptably.
  - quicksilver03 8y ago
    A million times this. I'm about to tackle this issue at my current gig, and I won't have any on-call rotation with less than 4 people.