4 ms·
> Every alert should be actionable. How do you know the difference between "everything is working properly" and "the logging and/or monitoring has stopped work
by AnIdiotOnTheNet 9y ago
> Every alert should be actionable.
How do you know the difference between "everything is working properly" and "the logging and/or monitoring has stopped working"?
- dewey 9y agoYou instrument your code instead of just logging. To see that your metric export is working you can regularly export a simple value and check for it’s existence.
- heegemcgee 9y agoWho will watch the watchmen, right? :D It's a real concern. Personally, i have a monitoring agent running, and then i have the config management agent (puppet) validate that the monitoring agent is running. And what Dewey said is absolutely right - you can monitor the code / service itself through health checks. In the case of a reports service, perhaps your monitor asks the API for a very small report. Or you could implement a special endpoint / controller that calls on the core code. I recently implemented a monitor that emulates a typical user session, logging in, performing popular tasks, and logging out. If any step in that process has an error, i get an alert with the step listed, and i instantly have some idea of where things are jammed. In this manner, i don't need to have pre-defined log monitors for specific errors; i can catch novel error types by virtue of exercising the code and watching for the expected responses - 200 in the header, ability to perform tasks that are only available on login, checking for certain strings in the response, etc.