3 ms·

Why do you need logs for that though? You could also use metrics and define the alert routing based on thread holds and alerting rules there. Also has the advan

by dewey 3y ago

Why do you need logs for that though? You could also use metrics and define the alert routing based on thread holds and alerting rules there. Also has the advantage that you can alert before something bad happens by looking at how a time series develops (like for example disk space)

Groxx 3y ago
Once you go down this path though, what do you need any logs for? Logs are standard and easy, which is why people use them.
- dewey 3y ago
  It is still very helpful to see for example which id or entity caused an issue or was slow. These are things you can’t feed into metrics usually. Metrics show you that something is wrong / alerting. Logs show you what is wrong and where to start debugging.
- otabdeveloper4 3y ago
  Realistically, you only need logs to debug crashes. (Segv's and null pointer violations and such.) Anything else will quickly get turned into a metric stream anyways.
  - Groxx 3y ago
    In terms of using stderr/stdout: yeah I think this is totally defensible. And depending on how hard you squint, and if you use Plan9, logging to a separate log file is just a metrics stream too, and it very well could be transformed into one before being sent anywhere else. This is overwhelmingly normal too, so it's not a hypothetical extreme at all - avoiding std* is often a good practice. Semi-structured streams with basically unbounded cardinality are still useful tho, and I think that's arguably "a log" regardless of how it's represented. And then levels come back into the system.
chipdart 3y ago
> Why do you need logs for that though? You could also use metrics and define the alert routing based on thread holds and alerting rules there. No, you cannot. At least for most of the practical cases. Some upstream dependencies can and do throw critical/fault-level log messages which you do not control and cannot catch. This scenario is frequent enough that some cloud providers even provide services that generate metrics streams by filtering logs. Also important: the operational side of running a service. When the shit hits the fan, you do not have the luxury of redeploying production services just because you need to add a log call or emit a metric. For that, you work with what you already have in place. I was involved in a couple of SEVs where I personally created metrics filters and alarms on the fly to help troubleshoot issues and monitor relevant failure modes.
eithed 3y ago
> You could also use metrics and define the alert routing based on thread holds and alerting rules there. Sure, if you want to setup such rules. Do you want to go through your service and define on the notification level that not every ERROR is CRITICAL?
- dewey 3y ago
  You write alerts based on a few high level metrics, and then define where these alerts are router and if they are critical vs. bad but don’t need to page someone.

by dewey 3y ago

Groxx 3y ago

Once you go down this path though, what do you need any logs for? Logs are standard and easy, which is why people use them.

dewey 3y ago
It is still very helpful to see for example which id or entity caused an issue or was slow. These are things you can’t feed into metrics usually. Metrics show you that something is wrong / alerting. Logs show you what is wrong and where to start debugging.
otabdeveloper4 3y ago
Realistically, you only need logs to debug crashes. (Segv's and null pointer violations and such.) Anything else will quickly get turned into a metric stream anyways.
- Groxx 3y ago
  In terms of using stderr/stdout: yeah I think this is totally defensible. And depending on how hard you squint, and if you use Plan9, logging to a separate log file is just a metrics stream too, and it very well could be transformed into one before being sent anywhere else. This is overwhelmingly normal too, so it's not a hypothetical extreme at all - avoiding std* is often a good practice. Semi-structured streams with basically unbounded cardinality are still useful tho, and I think that's arguably "a log" regardless of how it's represented. And then levels come back into the system.

chipdart 3y ago

> Why do you need logs for that though? You could also use metrics and define the alert routing based on thread holds and alerting rules there. No, you cannot. At least for most of the practical cases. Some upstream dependencies can and do throw critical/fault-level log messages which you do not control and cannot catch. This scenario is frequent enough that some cloud providers even provide services that generate metrics streams by filtering logs. Also important: the operational side of running a service. When the shit hits the fan, you do not have the luxury of redeploying production services just because you need to add a log call or emit a metric. For that, you work with what you already have in place. I was involved in a couple of SEVs where I personally created metrics filters and alarms on the fly to help troubleshoot issues and monitor relevant failure modes.

eithed 3y ago

> You could also use metrics and define the alert routing based on thread holds and alerting rules there. Sure, if you want to setup such rules. Do you want to go through your service and define on the notification level that not every ERROR is CRITICAL?

dewey 3y ago
You write alerts based on a few high level metrics, and then define where these alerts are router and if they are critical vs. bad but don’t need to page someone.