2 ms·

> Logs are independent. When you can not store every event, you can drop them randomly. You lose a perfect view of every logged event, but you still retain a st

by growse 2y ago

> Logs are independent. When you can not store every event, you can drop them randomly. You lose a perfect view of every logged event, but you still retain a statistical view. As we have already assumed you can not log everything, this is the best you can do anyways. Usually, every log message occurs as part of a process (no matter how short lived), and so I'm not sure it's ever the case that a given log is truly independent from others. "Sampling log events" is a smell that indicates that you will have incomplete views on any given txn/process. What I find works better is to always correlate logs to traces, and when just drop the majority of non-error traces (keep 100% of actionable error traces). > When you can not store every event, you want to drop randomly at the whole trace level. Yes, this.

0x6c6f6c 2y ago
I wonder if anyone has utilized trace capabilities like head and tail based sampling _for logging_. You could potentially have the same logic used to determine that logs do not need to be generated because there are no transaction logs ahead of it or before. Useful capabilities for tracing today that could translate well for logs too. So long as everthing is stitched together, which is what these tracing libraries are responsible for.
- phillipcarter 2y ago
  I'm not aware of tail-based (i.e., gathering a batch of logs and sampling the group coherently), but some of the sampling techniques mentioned in the post are absolutely used for structured logging systems. At least with Honeycomb, for logs unassociated with traces, you can use dynamic sampling[0] to significantly cut down on total event volume and bake in good representivity. And when you have logs correlated with traces, like how OTel does it, you can sample all logs correlated to a trace when that trace is also sampled[1]. We've at least also been doing some thinking about what batching up and coherently sampling "likely related but not correlated by ID" groups of logs. It's definitely in the realm of doable and TBH I find it a little strange that people haven't sought these kinds of solutions at large yet. I think most folks still just assume that the only way you can reduce volume is by giving up some notion of representativeness when that's just not true. [0]: https://github.com/honeycombio/dynsampler-go https://github.com/honeycombio/dynsampler-go [1]: https://github.com/honeycombio/refinery https://github.com/honeycombio/refinery
thayne 2y ago
Yes, you can probably combine traces and logs into a single category, and really what are traces but a collection of related logs with information about the relationship between them. But metrics are an entirely different beast. Sure some metrics can be derived from "events". But what about thinks like CPU and RAM usage? Do you want to record an event for every packet so you can measure network performance? And as mentioned above, you generally don't want to sample the things you have metrics if.
- growse 2y ago
  If I were being pedantic, I'd say that the metric of "CPU usage" comes from the event "measuring the CPU usage" :p
- yencabulator 2y ago
  > you generally don't want to sample the things you have metrics [o]f. Metrics are a sampling of a counter/gauge.

by growse 2y ago

0x6c6f6c 2y ago

I wonder if anyone has utilized trace capabilities like head and tail based sampling _for logging_. You could potentially have the same logic used to determine that logs do not need to be generated because there are no transaction logs ahead of it or before. Useful capabilities for tracing today that could translate well for logs too. So long as everthing is stitched together, which is what these tracing libraries are responsible for.

phillipcarter 2y ago
I'm not aware of tail-based (i.e., gathering a batch of logs and sampling the group coherently), but some of the sampling techniques mentioned in the post are absolutely used for structured logging systems. At least with Honeycomb, for logs unassociated with traces, you can use dynamic sampling[0] to significantly cut down on total event volume and bake in good representivity. And when you have logs correlated with traces, like how OTel does it, you can sample all logs correlated to a trace when that trace is also sampled[1]. We've at least also been doing some thinking about what batching up and coherently sampling "likely related but not correlated by ID" groups of logs. It's definitely in the realm of doable and TBH I find it a little strange that people haven't sought these kinds of solutions at large yet. I think most folks still just assume that the only way you can reduce volume is by giving up some notion of representativeness when that's just not true. [0]: https://github.com/honeycombio/dynsampler-go https://github.com/honeycombio/dynsampler-go [1]: https://github.com/honeycombio/refinery https://github.com/honeycombio/refinery

thayne 2y ago

Yes, you can probably combine traces and logs into a single category, and really what are traces but a collection of related logs with information about the relationship between them. But metrics are an entirely different beast. Sure some metrics can be derived from "events". But what about thinks like CPU and RAM usage? Do you want to record an event for every packet so you can measure network performance? And as mentioned above, you generally don't want to sample the things you have metrics if.

growse 2y ago
If I were being pedantic, I'd say that the metric of "CPU usage" comes from the event "measuring the CPU usage" :p
yencabulator 2y ago
> you generally don't want to sample the things you have metrics [o]f. Metrics are a sampling of a counter/gauge.