4 ms·
The premise is great: you should indeed not rely (only) on complex log and metrics solutions for actionable alerting. The solution is bad: you should not build
by trabant00 5y ago
The premise is great: you should indeed not rely (only) on complex log and metrics solutions for actionable alerting.
The solution is bad: you should not build a monitoring system yourself. Nagios family of monitoring systems have built a number of concepts from years of experience that you will not cover with DIY. Just a few examples not to make this huge:
- host and service dependencies. When a route goes down you do not want to receive hundreds of alerts for everything behind that route.
- health soft and hard state. A temporary (seconds) bad state should not send an alert. Only if the check fails several times in a row.
- state flaping detection. You do not want tens of alerts for something that becomes healthy and unhealthy several times over a short period.
- escalation: display in dashboard, after a time message L1 on chat, after a time notify L2 with SMS, after a time call over phone L3.
- support custom notification methods: for the above example be able to call by phone. Phone services are orders of magnitude more reliable than mail or http api calls.
- 4 service states: ok, warning, critical, unknown. Takes a lot to explain this but 2 states are not enough. For a networking outage you want things behind the failed point to be marked unknown as they might be ok for a customer taking another route to them than your monitoring.
- support SNMP among other things to be able to monitor network appliances.
I could go on but typing on the phone is not fun and I think this is enough for my point. Learn all things Nagios family of monitoring systems have to offer. Then decide if you can really do better with any other solution (you can, it's just not trivial though).
- sgt 5y agoWe use CheckMK for that. Works great after all these years. Complemented with log / API monitoring and dashboards.
- mobb_solo 5y agoMakes sense to me. However, I was still inspired by this writeup / approach. Specifically for hobbiest projects. I also think for the time being, I may implement this on my lame closet server.