3 ms·
That's exactly what happened though and no one bothered mentioning this to this reporter. I was in the post mortem for this event, the alert for database disk
by rrix2 9y ago
That's exactly what happened though and no one bothered mentioning this to this reporter.
I was in the post mortem for this event, the alert for database disk space was piled in to 500 other alerts incorrectly assigned to a rotation notorious for being the most painful in the company. The alerts escalated through to an infrastructure manager and then to an at-the-time director of engineering, none of whom acknowledged the alert themselves.
The alert itself was also of the "disk is 90% full" variety rather than "at rate of growth the disk will be full in an hour and by the way this is the master db". The former wouldn't be a need to panic as you'd have at least a day to resolve it, unless someone is back filling large sets of data.