5 ms·
I work on a large SaaS product that is deployed in AWS globally and is expected to have 100% uptime. Each service team in the SaaS has its own on call rotation
by iends 9y ago
I work on a large SaaS product that is deployed in AWS globally and is expected to have 100% uptime. Each service team in the SaaS has its own on call rotation and is responsible for being highly available. There are also architects and more DevOps folks (SRE team) always on call so you can always get help at 2am if you really need it.
I'm the primary on call every 3 weeks, but secondary the week after. Technically, since I'm the team lead, I'm always on call if the issue escalates to me at level 3. In practice my team is very good and it's extremely rare to get a page when you are not primary.
The team doesn't get paid extra, but it is mentioned in the interview completely up front and my company's compensation is pretty solid.
If paged after hours you are expected to work remote. If you can't get to a computer in 30 minutes you should escalate the issue (or just ignore the page). This is company policy, but I expect most companies to have some type of SLA for responding to pages.
As far as what type of pages we get...when there was flooding in AWS Australia a few months back I got paged because we lost a certain number of AWS nodes. Sometimes we deploy code with a memory leak and we find out a day or two later. Often, AWS just does something stupid (like the AWS S3 outage a few months back). Sometimes the DB cluster loses a node. Sometimes we ship code that interacts with another service in unintended ways. We NEVER interact with customers, all our pages are in response to specific technical issues. If customers need to be contacted there are people with customer service skills that convey the message appropriately. We do not get paged out for issues that are not major (e.g. the product or a major feature stops working).