4 ms·

I think outside some FAANG firms with strong engineering management, the purchase of these tools is a top-down affair. So as cynical as my statement may seem-

I think outside some FAANG firms with strong engineering management, the purchase of these tools is a top-down affair. So as cynical as my statement may seem- " It's almost like they wanted it to be easier to just not take time off?" .. user ergonomics just doesn't enter into the conversation at all, because the users are internal devs.. who cares. I also worked at an org that gave each of 50+ large customers a dedicated slack channel with.. wait for it.. the entire internal 200+ member engineering org in channel with them. You can imagine the mayhem as each customer figured out how to get attention (@here/@channel/@theguythatfixedthislasttime) and "hack the phone tree" as I used to call it. This also meant the every one of our 200+ engineers was a member of 50+ customer channels which were filled with all levels of noise that 95% of the time had nothing to do with their app/team/function, and could/should be ignored.

danpalmer 4y ago
> the purchase of these tools is a top-down affair This is absolutely it. Pager Duty don't sell on-call tools to engineers, they sell the idea of having an on-call rota to CTOs, the tools are an implementation detail. Bottom up engineering tools take more time to start with, but almost always cost less in the long run, contribute to a good engineering culture, and build a sense of ownership.
- thayne 4y ago
  That isn't universally true. At my medium sized company, the people on call are the ones responsible for choosing an on-call system. But we've rotated through most of the main options, because we haven't really been happy with any of them.
HereBeBeasties 4y ago
I am in management. I evaluated most of the on-duty tools that are available, at the cost of a couple of days of my personal time. At least they all have free trials. PageDuty, Opsgenie, Splunk On-Call (ex VictorOps), AlertOps, Squadcast, TaskCall, etc. etc. They are all surprisingly hopeless. For example, none of them have SCIM integration to make managing your teams in them automatic. All of them have clunky calendar overrides. None of them seem to integrate well with Outlook / Google Calendar, particularly none take into account holidays. Many have no Terraform provider to manage them, and the ones that do are clunky at best, and hard to set-up/manage. OIDC is hit-and-miss. For example, for PagerDuty you need to call up their support team and get them to manually tweak something that's not in the UI settings to get OIDC for Azure AD sign-ins to work. It's not that management is apathetic. We genuinely don't want to engineers spending their time working around vendor inadequacies and lashing this stuff together with barely-maintained scripting that they resent having to write in the first place. Why would anyone want that? Given there's seemingly no product out there that lets you avoid that, what should we do? When they're all rubbish, you either choose PagerDuty because everyone does, or Opsgenie as a protest vote, because at least both have Terraform providers and plug-ins for other things like Slack and Sentry, etc.
- steveBK123 4y ago
  My point is not that management is bad & doesn't care, but that management has different priorities & cares about different things than the people using the app. All of us have talked on this thread about all the things PD is bad at, but few have come to the defense with a list of things its great at. The sales pitch, I am sure, to management is more along the lines of - metrics, dashboards, reports, "accountability", response times, yada yada.. and because management write the checks, I'm sure thats an area they do deliver.

danpalmer 4y ago

> the purchase of these tools is a top-down affair This is absolutely it. Pager Duty don't sell on-call tools to engineers, they sell the idea of having an on-call rota to CTOs, the tools are an implementation detail. Bottom up engineering tools take more time to start with, but almost always cost less in the long run, contribute to a good engineering culture, and build a sense of ownership.

thayne 4y ago
That isn't universally true. At my medium sized company, the people on call are the ones responsible for choosing an on-call system. But we've rotated through most of the main options, because we haven't really been happy with any of them.

HereBeBeasties 4y ago

I am in management. I evaluated most of the on-duty tools that are available, at the cost of a couple of days of my personal time. At least they all have free trials. PageDuty, Opsgenie, Splunk On-Call (ex VictorOps), AlertOps, Squadcast, TaskCall, etc. etc. They are all surprisingly hopeless. For example, none of them have SCIM integration to make managing your teams in them automatic. All of them have clunky calendar overrides. None of them seem to integrate well with Outlook / Google Calendar, particularly none take into account holidays. Many have no Terraform provider to manage them, and the ones that do are clunky at best, and hard to set-up/manage. OIDC is hit-and-miss. For example, for PagerDuty you need to call up their support team and get them to manually tweak something that's not in the UI settings to get OIDC for Azure AD sign-ins to work. It's not that management is apathetic. We genuinely don't want to engineers spending their time working around vendor inadequacies and lashing this stuff together with barely-maintained scripting that they resent having to write in the first place. Why would anyone want that? Given there's seemingly no product out there that lets you avoid that, what should we do? When they're all rubbish, you either choose PagerDuty because everyone does, or Opsgenie as a protest vote, because at least both have Terraform providers and plug-ins for other things like Slack and Sentry, etc.

steveBK123 4y ago
My point is not that management is bad & doesn't care, but that management has different priorities & cares about different things than the people using the app. All of us have talked on this thread about all the things PD is bad at, but few have come to the defense with a list of things its great at. The sales pitch, I am sure, to management is more along the lines of - metrics, dashboards, reports, "accountability", response times, yada yada.. and because management write the checks, I'm sure thats an area they do deliver.