3 ms·

I feel like a lot of the complaint here is why logging systems typically have levels? I use the following philosophy. Typically PROD would run at WARNING or m

I feel like a lot of the complaint here is why logging systems typically have levels? I use the following philosophy. Typically PROD would run at WARNING or maybe INFO, and never at DEBUG unless there's something happening that we can't figure out how to reproduce locally. DEBUG is really for local analysis, in which case you don't really care about overlogging. While ERROR level logging should immediately generate alerts, it's still useful to watch metrics on the WARNING level to see if something is potentially heading towards ERROR-level failure. ERROR / FATAL: The application failed to execute the business operation. WARNING: The application had an issue executing the business operation, but has a fallback -- for instance, retrying. Might still result in an ERROR later, if the fallback operation fails. INFO: Document the successful exit of business operations and decisions. This typically helps you determine where in the process the failure occurred, and what might need to be repaired if you decide to manually rollback the operation. DEBUG: Document the entrance of business operations and decisions, along with any additional technical operations, decisions, and values that might be relevant.

fatnoah 5y ago
In a past life, I spent a lot of time aligning and instrumenting log levels across a SaaS product. We used similar levels. DEBUG - Only useful for developers and QA looking to understand exactly what happened and why on a per-request basis. INFO - Provide information about the current state of the system. This was our production Log Level WARNING - Something happened that will likely require action at some point in the near future, but not immediately. i.e. Approaching quota limits, temporary conditions we can work around, etc. ERROR - Something unexpected happened and should be looked at soon if only a one-off or immediately if consistent. SEV 1 or 2 depending. FATAL - Conditions prevent the system from being able to operate correctly. Generally a complete and catastrophic failure that will not resolve on its own, such as missing dependencies, invalid configuration, etc.
015a 5y ago
This makes sense, in a "theory" sort of way. The reality I've far too often ran into is: something is logged as debug, production is set to info+, something rare happens, and we don't have the logging to diagnose it because it was being logged at debug. So we swap it out for an info log, or raise the global logging level of production, but we still can't diagnose it because the rare event happened in the past; we just hope it happens again, or we spend hours trying to blindly figure out the conditions which caused it to happen without the information to actually reproduce. Eventually management gets fed up, and authorizes debug+ ingestion. The logging bill doubles, but at least we can "solve it" with money (the log stream is twice as noisy, and half as performant, but at least we have the data right?) A lower-level problem surfacing here: As engineers, we decide on a log level when we write the code. Its in response to expected system behavior; not actual system behavior. In comparison, a better solution would be: surface logs only when behavior betrays expectation. I don't want that debug line in 99% of requests, but for a 5xx request? I do. Logging systems are horribly unsuited for this, because it requires (1) a context (the request), (2) correlation between different execution points in the context (only log if later the request ends in 5xx), and (3) delaying log output (or ingestion) until that correlation is completed. We have a solution which does this out-of-the-box: Tracing. Tracing is designed to be sampled, and trace ingestion systems all have a "sampling" function designed to identify and surface only "interesting" or "anomalous" traces (if you want).

fatnoah 5y ago

In a past life, I spent a lot of time aligning and instrumenting log levels across a SaaS product. We used similar levels. DEBUG - Only useful for developers and QA looking to understand exactly what happened and why on a per-request basis. INFO - Provide information about the current state of the system. This was our production Log Level WARNING - Something happened that will likely require action at some point in the near future, but not immediately. i.e. Approaching quota limits, temporary conditions we can work around, etc. ERROR - Something unexpected happened and should be looked at soon if only a one-off or immediately if consistent. SEV 1 or 2 depending. FATAL - Conditions prevent the system from being able to operate correctly. Generally a complete and catastrophic failure that will not resolve on its own, such as missing dependencies, invalid configuration, etc.

015a 5y ago

This makes sense, in a "theory" sort of way. The reality I've far too often ran into is: something is logged as debug, production is set to info+, something rare happens, and we don't have the logging to diagnose it because it was being logged at debug. So we swap it out for an info log, or raise the global logging level of production, but we still can't diagnose it because the rare event happened in the past; we just hope it happens again, or we spend hours trying to blindly figure out the conditions which caused it to happen without the information to actually reproduce. Eventually management gets fed up, and authorizes debug+ ingestion. The logging bill doubles, but at least we can "solve it" with money (the log stream is twice as noisy, and half as performant, but at least we have the data right?) A lower-level problem surfacing here: As engineers, we decide on a log level when we write the code. Its in response to expected system behavior; not actual system behavior. In comparison, a better solution would be: surface logs only when behavior betrays expectation. I don't want that debug line in 99% of requests, but for a 5xx request? I do. Logging systems are horribly unsuited for this, because it requires (1) a context (the request), (2) correlation between different execution points in the context (only log if later the request ends in 5xx), and (3) delaying log output (or ingestion) until that correlation is completed. We have a solution which does this out-of-the-box: Tracing. Tracing is designed to be sampled, and trace ingestion systems all have a "sampling" function designed to identify and surface only "interesting" or "anomalous" traces (if you want).