4 ms·
"The temperature is on a trajectory to breach the machine's operating limit in N minutes" seems pretty actionable. I follow Google's symptom-based alerting phi
by closeparen 7y ago
"The temperature is on a trajectory to breach the machine's operating limit in N minutes" seems pretty actionable.
I follow Google's symptom-based alerting philosophy in general, but will make an exception when there's a chance to catch something getting dangerously close to an unambiguous hard-failure limit (e.g. 90% quota utilization).
- thethethethe 7y ago> "The temperature is on a trajectory to breach the machine's operating limit in N minutes" seems pretty actionable. Do you really think that the folks in the Google datacenters are not monitoring rack temp? Implemnting Symptom based alerting does not mean you should not be monitoring other system metrics. Their monitoring system probably filed a P1 or P2 ticket for someone to go take a look at it at some point. But should a person be paged at 2 A.M to repair this? Absolutely not.
- joshuamorton 7y ago> "The temperature is on a trajectory to breach the machine's operating limit in N minutes" seems pretty actionable. For who? For an application, the job will just get rescheduled onto a different machine, and you're N+2 or whatever so no one will notice the restart. For the datacenter, since (for most jobs) you can assume that no one will care, you don't necessarily monitor for single machines overheating, but rack or row level issues, if a machine keeps overheating, eventually "nothing runs on this machine for more than an hour without it restarting" -symptom based alerting kicks in and tells someone to look at the machine.