3 ms·

> Logging events are the epitome of observability, and metrics and tracing events end up specializations of logging events which only cover very precise and spe

> Logging events are the epitome of observability, and metrics and tracing events end up specializations of logging events which only cover very precise and specific usecase. This is a common way of thinking about things, but having now worked with event-based tracing for the last few years I no longer believe traditional logs are necessary. If you do “good” logging you likely have entirely structured data and no dynamic portions of the main log message. This is almost exactly equivalent to a span as represented by tracing implementations. I have come to believe that these events are the core piece of telemetry, and from them you can derive metrics, spans, traces and even normal-looking logs.

arinlen 4y ago
> This is a common way of thinking about things, but having now worked with event-based tracing for the last few years I no longer believe traditional logs are necessary. You are free to use stuff as you see fit, even if you intentionally miss out on features. Logging is a very basic feature whose usecases are not covered by metrics events, or even traces. Logs are used to help developers troubleshoot problems by giving them an open format to output data that's relevant to monitor specific aspects of a system's behavior. It makes no sense to emit a metrics event with tons of stuff as dimensions and/or annotations that's emitted in a single line of code that's executed by a single code path when all you need is a single log message. Also, stack traces are emitted as logging events, and metrics events are pretty useless at tracking that sort of info. But hey, to each its own. If you feel that emitting a counter or a timer gives you enough info to check why your code caught an exception and what made it crash, more power to you. The whole world seems to disagree, given standard logging frameworks even added support for logging stack traces as structured logs as a basic feature.
- glenjamin 4y ago
  A counter or a timer is clearly no substitute for a log, that would be an absurd statement for me to have made - which is why I said no such thing. A span as modelled by OpenTelemetry is a a bag of developer-chosen structured data containing a name, a timestamp and some auto-generated IDs to correlate it with others. It is effectively an open standard for how to collect good structured logs - although it is currently not commonly pitched that way by either tracing advocates or structured logging advocates. These are commonly used to create trace waterfalls, but can also be used to derive metrics or simple dev-debug output. I have used this approach myself to great effect, and also spoken to a number of people who have also found success with such an approach - but currently the momentum of existing approaches (and the fact that traditional logging still works well-enough for most people) has not led to wide adoption
wsargent 4y ago
I see it the other way around: tracing is essentially logging with hierarchy. If you can keep the context of a "parent" span around, then you can log all of the entries out and build up the entire trace from spans, although you can get pretty confused if you don't write them out in the correct order. However, if you don't have context, then the log entry is the atomic unit -- metrics are log entries with numbers that can be aggregated, spans are log entries with a trace id and a parent span, and "events" in Honeycomb terminology are logs with attributes covering an entire business operation / HTTP request.

arinlen 4y ago

> This is a common way of thinking about things, but having now worked with event-based tracing for the last few years I no longer believe traditional logs are necessary. You are free to use stuff as you see fit, even if you intentionally miss out on features. Logging is a very basic feature whose usecases are not covered by metrics events, or even traces. Logs are used to help developers troubleshoot problems by giving them an open format to output data that's relevant to monitor specific aspects of a system's behavior. It makes no sense to emit a metrics event with tons of stuff as dimensions and/or annotations that's emitted in a single line of code that's executed by a single code path when all you need is a single log message. Also, stack traces are emitted as logging events, and metrics events are pretty useless at tracking that sort of info. But hey, to each its own. If you feel that emitting a counter or a timer gives you enough info to check why your code caught an exception and what made it crash, more power to you. The whole world seems to disagree, given standard logging frameworks even added support for logging stack traces as structured logs as a basic feature.

glenjamin 4y ago
A counter or a timer is clearly no substitute for a log, that would be an absurd statement for me to have made - which is why I said no such thing. A span as modelled by OpenTelemetry is a a bag of developer-chosen structured data containing a name, a timestamp and some auto-generated IDs to correlate it with others. It is effectively an open standard for how to collect good structured logs - although it is currently not commonly pitched that way by either tracing advocates or structured logging advocates. These are commonly used to create trace waterfalls, but can also be used to derive metrics or simple dev-debug output. I have used this approach myself to great effect, and also spoken to a number of people who have also found success with such an approach - but currently the momentum of existing approaches (and the fact that traditional logging still works well-enough for most people) has not led to wide adoption

wsargent 4y ago

I see it the other way around: tracing is essentially logging with hierarchy. If you can keep the context of a "parent" span around, then you can log all of the entries out and build up the entire trace from spans, although you can get pretty confused if you don't write them out in the correct order. However, if you don't have context, then the log entry is the atomic unit -- metrics are log entries with numbers that can be aggregated, spans are log entries with a trace id and a parent span, and "events" in Honeycomb terminology are logs with attributes covering an entire business operation / HTTP request.