3 ms·

You absolutely DO want all of those log levels. Log channels allow you to route certain messages to certain systems. CRITICAL, for example, can be forwarded to

You absolutely DO want all of those log levels. Log channels allow you to route certain messages to certain systems. CRITICAL, for example, can be forwarded to your "you have to deal with this RIGHT NOW" alert system (CRITICAL: Stock trading algo lost 75% of its portfolio value in the last hour). That's very different from a regular (ERROR: Endpoint /report responded with code 503). DEBUG and TRACE can be combined with dynamic filters to keep the volume of messages down while still being able to get useful information from a production system. There are also dev and staging servers for running general DEBUG and TRACE in a lower volume environment.

deaddodo 3y ago
I would argue that the only three I care about in production software are INFO, ERROR (an issue occurred but the system is still running) and CRITICAL (an error occurred rendering a crash). Any others are only really useful for debug situations.
- chopin 3y ago
  I had errors which occurred only in production or specific users. The ability to even log at debug level in production is crucial.
aaron695 3y ago
[dead]
dewey 3y ago
Why do you need logs for that though? You could also use metrics and define the alert routing based on thread holds and alerting rules there. Also has the advantage that you can alert before something bad happens by looking at how a time series develops (like for example disk space)
- Groxx 3y ago
  Once you go down this path though, what do you need any logs for? Logs are standard and easy, which is why people use them.
  - dewey 3y ago
    It is still very helpful to see for example which id or entity caused an issue or was slow. These are things you can’t feed into metrics usually. Metrics show you that something is wrong / alerting. Logs show you what is wrong and where to start debugging.
  - otabdeveloper4 3y ago
    Realistically, you only need logs to debug crashes. (Segv's and null pointer violations and such.) Anything else will quickly get turned into a metric stream anyways.
    - Groxx 3y ago
      In terms of using stderr/stdout: yeah I think this is totally defensible. And depending on how hard you squint, and if you use Plan9, logging to a separate log file is just a metrics stream too, and it very well could be transformed into one before being sent anywhere else. This is overwhelmingly normal too, so it's not a hypothetical extreme at all - avoiding std* is often a good practice. Semi-structured streams with basically unbounded cardinality are still useful tho, and I think that's arguably "a log" regardless of how it's represented. And then levels come back into the system.
- chipdart 3y ago
  > Why do you need logs for that though? You could also use metrics and define the alert routing based on thread holds and alerting rules there. No, you cannot. At least for most of the practical cases. Some upstream dependencies can and do throw critical/fault-level log messages which you do not control and cannot catch. This scenario is frequent enough that some cloud providers even provide services that generate metrics streams by filtering logs. Also important: the operational side of running a service. When the shit hits the fan, you do not have the luxury of redeploying production services just because you need to add a log call or emit a metric. For that, you work with what you already have in place. I was involved in a couple of SEVs where I personally created metrics filters and alarms on the fly to help troubleshoot issues and monitor relevant failure modes.
- eithed 3y ago
  > You could also use metrics and define the alert routing based on thread holds and alerting rules there. Sure, if you want to setup such rules. Do you want to go through your service and define on the notification level that not every ERROR is CRITICAL?
  - dewey 3y ago
    You write alerts based on a few high level metrics, and then define where these alerts are router and if they are critical vs. bad but don’t need to page someone.
chipdart 3y ago
> You absolutely DO want all of those log levels. I agree, this is a very clear fact that's readily understood by anyone with any experience developing and operating a service. It's weird that we see claims arguing one of the core tools of troubleshooting isn't needed, without providing any sound argument. Among the long list of arguments that can be presented, I'm only going to point out one which might be obscure: cost. Cloud providers charge a premium for data generated through log ingestion, and the more logs you generate the more you pay. Most of the time needlessly so. One strategy to work around that is only logging error/critical in production. That avoids the invoice caused by all other log levels, which dominate the whole event dump. However, when the shit hits the fan you will want as much info as you can to help you quickly troubleshoot problems. With the standard log level hierarchy that's as easy as tweaking an environment variable in prod to output additional log levels. If you want to, you can drive the level down to dumping tracing info. Modern production-ready logging frameworks even support filtering log events based on the namespace/package/class level, for this very reason. You pepper your code paths with the relevant log messages corresponding to adequate log levels, and once you have your infrastructure sorted out you have all the tools you need to adapt your logs to the operational needs you have. You absolutely cannot do that with simplistic info/error logs.

deaddodo 3y ago

I would argue that the only three I care about in production software are INFO, ERROR (an issue occurred but the system is still running) and CRITICAL (an error occurred rendering a crash). Any others are only really useful for debug situations.

chopin 3y ago
I had errors which occurred only in production or specific users. The ability to even log at debug level in production is crucial.

aaron695 3y ago

[dead]

dewey 3y ago

Why do you need logs for that though? You could also use metrics and define the alert routing based on thread holds and alerting rules there. Also has the advantage that you can alert before something bad happens by looking at how a time series develops (like for example disk space)

Groxx 3y ago
Once you go down this path though, what do you need any logs for? Logs are standard and easy, which is why people use them.
- dewey 3y ago
  It is still very helpful to see for example which id or entity caused an issue or was slow. These are things you can’t feed into metrics usually. Metrics show you that something is wrong / alerting. Logs show you what is wrong and where to start debugging.
- otabdeveloper4 3y ago
  Realistically, you only need logs to debug crashes. (Segv's and null pointer violations and such.) Anything else will quickly get turned into a metric stream anyways.
  - Groxx 3y ago
    In terms of using stderr/stdout: yeah I think this is totally defensible. And depending on how hard you squint, and if you use Plan9, logging to a separate log file is just a metrics stream too, and it very well could be transformed into one before being sent anywhere else. This is overwhelmingly normal too, so it's not a hypothetical extreme at all - avoiding std* is often a good practice. Semi-structured streams with basically unbounded cardinality are still useful tho, and I think that's arguably "a log" regardless of how it's represented. And then levels come back into the system.
chipdart 3y ago
> Why do you need logs for that though? You could also use metrics and define the alert routing based on thread holds and alerting rules there. No, you cannot. At least for most of the practical cases. Some upstream dependencies can and do throw critical/fault-level log messages which you do not control and cannot catch. This scenario is frequent enough that some cloud providers even provide services that generate metrics streams by filtering logs. Also important: the operational side of running a service. When the shit hits the fan, you do not have the luxury of redeploying production services just because you need to add a log call or emit a metric. For that, you work with what you already have in place. I was involved in a couple of SEVs where I personally created metrics filters and alarms on the fly to help troubleshoot issues and monitor relevant failure modes.
eithed 3y ago
> You could also use metrics and define the alert routing based on thread holds and alerting rules there. Sure, if you want to setup such rules. Do you want to go through your service and define on the notification level that not every ERROR is CRITICAL?
- dewey 3y ago
  You write alerts based on a few high level metrics, and then define where these alerts are router and if they are critical vs. bad but don’t need to page someone.

chipdart 3y ago

> You absolutely DO want all of those log levels. I agree, this is a very clear fact that's readily understood by anyone with any experience developing and operating a service. It's weird that we see claims arguing one of the core tools of troubleshooting isn't needed, without providing any sound argument. Among the long list of arguments that can be presented, I'm only going to point out one which might be obscure: cost. Cloud providers charge a premium for data generated through log ingestion, and the more logs you generate the more you pay. Most of the time needlessly so. One strategy to work around that is only logging error/critical in production. That avoids the invoice caused by all other log levels, which dominate the whole event dump. However, when the shit hits the fan you will want as much info as you can to help you quickly troubleshoot problems. With the standard log level hierarchy that's as easy as tweaking an environment variable in prod to output additional log levels. If you want to, you can drive the level down to dumping tracing info. Modern production-ready logging frameworks even support filtering log events based on the namespace/package/class level, for this very reason. You pepper your code paths with the relevant log messages corresponding to adequate log levels, and once you have your infrastructure sorted out you have all the tools you need to adapt your logs to the operational needs you have. You absolutely cannot do that with simplistic info/error logs.