2 ms·
Largeish company (engineering staff around 1000, I think?), and my team owns the OS platform (packaging, config management), hardware purchasing, and the instal
by geofft 7y ago
Largeish company (engineering staff around 1000, I think?), and my team owns the OS platform (packaging, config management), hardware purchasing, and the installer, as well as our private cloud, which hosts a good chunk of services (notably including dev machines, notably excluding the most money-making parts of the business). A separate team is first-line sysadmin+SRE for most of the company, so we don't respond unless you can't install machines or if every machine is broken, or if the private cloud (as infrastructure, not a specific VM on it) is acting up.
Private cloud was originally a separate team, and we had about 5 people on a weekly rotation, where you'd do support and 24/7 on-call for a week. Now that we've merged, it's about a dozen people on the on-call rotation, though we still have a subteam of about 5 for the weekly rotation of who's paying attention to private cloud support tickets during working hours.
Thanks to some work that we did a bit after I joined, we got the off-hours paging to be much less frequent - in particular we no longer page if any hypervisor is down, only if either a whole availability zone is down, or a hypervisor hosting personal dev machines is down and it's business hours. (Otherwise, higher-level services ought to have failover between multiple VMs, and devs working odd hours can use backup machines.) Usually that second condition gets noticed by the support person (in Jira) before it escalates to the on-call person. So it's rare to have anyone actually paged, and it usually means something unexpected is on fire. So, any member of the team is as good as any other at figuring out what broke and whether it can wait until morning, which is why we were able to throw all of the combined team into the rotation without training them all in everything.
There are still improvements I want to do, but the expected number of pages rounds to zero (it looks like we've had 6 off-hours pages this year) so it's certainly manageable.