4 ms·

I can understand not doing butts in chairs 24/7, but I'm really surprised they managed to make it to that scale without adding paging escalation. If someone doe

by kevan 9y ago

I can understand not doing butts in chairs 24/7, but I'm really surprised they managed to make it to that scale without adding paging escalation. If someone doesn't respond after X minutes then page their manager. Repeat until you get to the CEO.

oh_sigh 9y ago
Yes....that's the way it is set up at Amazon, and I recall one time a page from my little teams project with no revenue or customer implications for downtime managed to make it all the way to one level below Bezos before it was acked (this was 1am after Thanksgiving, 2012).
jfoutz 9y ago
That was my initial thought as well. But, low disk space on the master DB? And you need a human to go futz around with it? There is either a whole lot of missing monitoring and automation, or those are completely worthless pages. I understand the freakout, i assume it could take down the site. But, like, not responding to a page is like the very last on the list of things to fix. Calling in random engineer is pretty much last ditch, hail mary, the world is on fire, oh fuck we're going to go out of business disaster. At a certain point, you kind of have to assume one possibility is your oncall person is dead. Whatcha gonna do then? I seriously doubt all of their careful high availability planning failed. I would bet their paging is (hopefully was) just stupid.
- oh_sigh 9y ago
  From the article, it's not that the engineer never responded. The engineer acknowledged the pages but never did anything to correct the problem beyond that. I agree with you though that a random engineer might not have the tools and knowledge ready to fix something like that, but that is why you can let the pages escalate until you wake someone up who does.
oh_sigh 9y ago
From the article, it is stated that the oncall engineer acked the pages for low disk space repeatedly but never took any corrective actions. So, escalation wasn't exercised in this case because the engineer tricked the system into thinking they handled it.
- kevan 9y ago
  Ah, so now it just sounds like a textbook case of alarm fatigue. That's a standard problem at every company in the world, this case just happened to cause a very high publicity outage.
  - oh_sigh 9y ago
    Yup - that is also in Pham's initial email as the face saving alternative to negligence
rrix2 9y ago
That's exactly what happened though and no one bothered mentioning this to this reporter. I was in the post mortem for this event, the alert for database disk space was piled in to 500 other alerts incorrectly assigned to a rotation notorious for being the most painful in the company. The alerts escalated through to an infrastructure manager and then to an at-the-time director of engineering, none of whom acknowledged the alert themselves. The alert itself was also of the "disk is 90% full" variety rather than "at rate of growth the disk will be full in an hour and by the way this is the master db". The former wouldn't be a need to panic as you'd have at least a day to resolve it, unless someone is back filling large sets of data.

by kevan 9y ago

oh_sigh 9y ago

Yes....that's the way it is set up at Amazon, and I recall one time a page from my little teams project with no revenue or customer implications for downtime managed to make it all the way to one level below Bezos before it was acked (this was 1am after Thanksgiving, 2012).

jfoutz 9y ago

That was my initial thought as well. But, low disk space on the master DB? And you need a human to go futz around with it? There is either a whole lot of missing monitoring and automation, or those are completely worthless pages. I understand the freakout, i assume it could take down the site. But, like, not responding to a page is like the very last on the list of things to fix. Calling in random engineer is pretty much last ditch, hail mary, the world is on fire, oh fuck we're going to go out of business disaster. At a certain point, you kind of have to assume one possibility is your oncall person is dead. Whatcha gonna do then? I seriously doubt all of their careful high availability planning failed. I would bet their paging is (hopefully was) just stupid.

oh_sigh 9y ago
From the article, it's not that the engineer never responded. The engineer acknowledged the pages but never did anything to correct the problem beyond that. I agree with you though that a random engineer might not have the tools and knowledge ready to fix something like that, but that is why you can let the pages escalate until you wake someone up who does.

oh_sigh 9y ago

From the article, it is stated that the oncall engineer acked the pages for low disk space repeatedly but never took any corrective actions. So, escalation wasn't exercised in this case because the engineer tricked the system into thinking they handled it.

kevan 9y ago
Ah, so now it just sounds like a textbook case of alarm fatigue. That's a standard problem at every company in the world, this case just happened to cause a very high publicity outage.
- oh_sigh 9y ago
  Yup - that is also in Pham's initial email as the face saving alternative to negligence

rrix2 9y ago

That's exactly what happened though and no one bothered mentioning this to this reporter. I was in the post mortem for this event, the alert for database disk space was piled in to 500 other alerts incorrectly assigned to a rotation notorious for being the most painful in the company. The alerts escalated through to an infrastructure manager and then to an at-the-time director of engineering, none of whom acknowledged the alert themselves. The alert itself was also of the "disk is 90% full" variety rather than "at rate of growth the disk will be full in an hour and by the way this is the master db". The former wouldn't be a need to panic as you'd have at least a day to resolve it, unless someone is back filling large sets of data.