4 ms·

There are both models at Amazon. Individual teams have their own on-call; if for example an RDS instance is continually failing to boot, someone on-call (edit:

by mjr00 4y ago

There are both models at Amazon. Individual teams have their own on-call; if for example an RDS instance is continually failing to boot, someone on-call (edit: from the RDS team) will get paged to look at it. I believe order drops are looked at by dedicated reliability team, because it's such a significant problem. But they don't watch graphs because... why would they? They set up the machines to page them when metrics aren't within an expected range. Which is what happened.

majormajor 4y ago
That makes a lot of sense. The ops teams I've worked with watched graphs because (a) sometimes you see interesting changes or trends before they turn into actionable alerts and people can look into them/figure them out before shit's on fire, and (b) it makes it look more like you're doing something when everything is smooth sailing and not actively on fire, solving the "wait why are we paying for this if it doesn't take us long to respond to incidents anyway" naive-meddlesome-exec problem.