This makes sense, in a "theory" sort of way. The reality I've far too often ran into is: something is logged as debug, production is set to info+, something ra

This makes sense, in a "theory" sort of way. The reality I've far too often ran into is: something is logged as debug, production is set to info+, something rare happens, and we don't have the logging to diagnose it because it was being logged at debug. So we swap it out for an info log, or raise the global logging level of production, but we still can't diagnose it because the rare event happened in the past; we just hope it happens again, or we spend hours trying to blindly figure out the conditions which caused it to happen without the information to actually reproduce. Eventually management gets fed up, and authorizes debug+ ingestion. The logging bill doubles, but at least we can "solve it" with money (the log stream is twice as noisy, and half as performant, but at least we have the data right?) A lower-level problem surfacing here: As engineers, we decide on a log level when we write the code. Its in response to expected system behavior; not actual system behavior. In comparison, a better solution would be: surface logs only when behavior betrays expectation. I don't want that debug line in 99% of requests, but for a 5xx request? I do. Logging systems are horribly unsuited for this, because it requires (1) a context (the request), (2) correlation between different execution points in the context (only log if later the request ends in 5xx), and (3) delaying log output (or ingestion) until that correlation is completed. We have a solution which does this out-of-the-box: Tracing. Tracing is designed to be sampled, and trace ingestion systems all have a "sampling" function designed to identify and surface only "interesting" or "anomalous" traces (if you want).