3 ms·
Google SRE here. Don't follow this article literally - as great as monitoring (which I take to mean metrics that you can use to plot graphs) is, you'll never h
by telotortium 5y ago
Google SRE here.
Don't follow this article literally - as great as monitoring (which I take to mean metrics that you can use to plot graphs) is, you'll never have defined as many monitoring metrics as you want, and logging can contain a much greater level of detail for each event, which allows you to reconstruct the course of an incident after the fact (allowing you to solve the incident much faster than trying to figure out how to reproduce it). Also, in regulated environments, logging has the advantage that it can contain data that's too sensitive to be recorded long-term, even with ACL checks, since debug logs are generally retained for only a short time (few days or so). You generally want your graphs to be able to use data from farther back than that (up to a year).
The best article that encompasses the Google logging philosophy is https://apenwarr.ca/log/20190216 https://apenwarr.ca/log/20190216
Other points:
- Don't log to a synchronized file if possible - log either to the network or at least to RAM (which can be emptied to the network in the background).
- For the overlogging problem, get a logging library that rate-limits your logging for DEBUG/INFO level events, ideally rate limiting by call site.
- For Java, it's very convenient to be able to instrument code at runtime to add and remove logpoints.