3 ms·
"proper monitoring" is extremely broad. And, I would say, almost unreachable goal. You have it mail you when it goes over 80% disk usage (and what if you are o
by berkes 6y ago
"proper monitoring" is extremely broad. And, I would say, almost unreachable goal.
You have it mail you when it goes over 80% disk usage (and what if you are on holiday)? Does it mail all colleagues? Who picks it up (I thought Bob picked it up, but Bob thought Anne picked it up. So no one did)? Does it come and wake you in person when it reaches 92%?
Will this catch this async job that fails (but should never) in an endless loop but keeps creating 20MB json files as fast as the disk allows it to?
Is it an alerting that finds anomalies in trends? Will it be fast enough for you to come online before that job has filled the disk?
I've been doing a lot of hosting management and such. And there is one constant: all unforeseen issues are unforeseen.
- geocrasher 6y ago> I've been doing a lot of hosting management and such. And there is one constant: all unforeseen issues are unforeseen. I work in hosting too, and have been for a long time. I feel ya.
- dilyevsky 6y agoSlack warning/ticket at 75%, page at 85% (to oncall obviously). Don’t let user workload crap into your root partition. I’ve been doing this for over 10 years and managed many thousands of nodes and literally don’t recall full disk problem unless it was in staging somewhere where monitoring was deliberately disabled.
- berkes 6y agoYour requirements for "proper monitoring" are not everyones requirements. On a current gig, we host at heroku. Our monitoring is all about 95th percentile response-times, secondary services, backlogs, slow-queries and whatnots. For another job, "disk space filling up" is important. Again another job will need to monitor email-delivery-rates and so on and so forth.