26 ms·

The isomorphism of traces and logs is clear. You can flatten a trace to a log and you can perfectly reconstruct the trace graph from such a log. I don't see the

The isomorphism of traces and logs is clear. You can flatten a trace to a log and you can perfectly reconstruct the trace graph from such a log. I don't see the unifying theme that brings metrics into this framework, though. Metrics feels fundamentally different, as a way to inspect the internal state of your program, not necessarily driven by exogenous events. But I definitely agree with the theme of the article that leaving a big company can feel like you got your memory erased in a time machine mishap. Inside a FANG you might become normalized to logging hundreds of thousands of informational statements, per second, per core. You might have got used to every endpoint exposing thirty million metric time series. As soon as you walk out the door some guy will chew you out about "cardinality" if you have 100 metrics.

HeyImAlex 3y ago
I think all metrics can be reconstructed as “wide events” since they’re just a bunch of arbitrary data? Counts, gauges, and histograms at least seem pretty straight forward to me. It seems like the main motivation for metrics is that sending + storing + querying wide events for everything is cost prohibitive and/or performance intensive. If you can afford it and it works well, wide events is definitely more flexible. A metric is kinda just a pre-aggregation on the event stream.
growse 3y ago
If you think of a metric as an event representing the act of measuring (along with the result of that measurement), then it becomes the same as any other event.
- jeffbee 3y ago
  True. I guess the thing that I normally want from metrics is I want to have a huge number of them that exist in a way that I can look at them when I want. But I don't want to have to pay for collecting and aggregating them all the time. So in the scenario where they are just events then I need some other control system that can trigger the collection of events that aren't normally emitted
  - growse 3y ago
    With metrics, you're always sampling. It's impossible to know the value of the measurement at every point in time. When you collect any form of metrics, something is choosing that sample rate.
    - jeffbee 3y ago
      That's not true in all models. For example in the (execrable) `statsd` model there is a bit of information sent every time a metric changes.
      - deathanatos 3y ago
        It's not just the stats protocol, it's the underlying metric, too. statsd is just a way of recording/transmitting metrics. If I transmit a statsd metric representing "CPU usage", I am still sampling it. E.g., I might read the CPU usage every second & generate a statsd stat. That's a sample rate of 1Hz. I have to choose some sampling frequency, since the API most OS's expose is "what's the current CPU usage?". If the metric is "total number of HTTP requests", then I can definitely just transmit that metric every time I get a request. We're not sampling for that metric. The latter is inherently a discrete event, with which we can know every data point of, though. Things like CPU, memory, are either fundamentally continuous, or their implementations are simply sampling it. I do agree the model matters too; Prom's tendency to just poll /metrics endpoints every n seconds means even things like HTTP events are inherently sampled.
        ekimekim 3y ago
        > If I transmit a statsd metric representing "CPU usage", I am still sampling it. In practice this is how everyone does it, but in theory it should be possible to have a non-sampled view of CPU usage (defined as "time process is scheduled onto a CPU"). With the right kernel introspection, you could represent it as a series of spans covering each time slice where the process is scheduled. Perhaps with a concept of a "currently ongoing span" to account for the current time slice. Do I think this would be more useful than the typical sampled metric? Probably not, outside some niche performance analysis workflows. But my point is that CPU is not actually continuous, and I struggle to think of any metric which cannot be represented without sampling if you REALLY need it.

HeyImAlex 3y ago

I think all metrics can be reconstructed as “wide events” since they’re just a bunch of arbitrary data? Counts, gauges, and histograms at least seem pretty straight forward to me. It seems like the main motivation for metrics is that sending + storing + querying wide events for everything is cost prohibitive and/or performance intensive. If you can afford it and it works well, wide events is definitely more flexible. A metric is kinda just a pre-aggregation on the event stream.

growse 3y ago

If you think of a metric as an event representing the act of measuring (along with the result of that measurement), then it becomes the same as any other event.

jeffbee 3y ago
True. I guess the thing that I normally want from metrics is I want to have a huge number of them that exist in a way that I can look at them when I want. But I don't want to have to pay for collecting and aggregating them all the time. So in the scenario where they are just events then I need some other control system that can trigger the collection of events that aren't normally emitted
- growse 3y ago
  With metrics, you're always sampling. It's impossible to know the value of the measurement at every point in time. When you collect any form of metrics, something is choosing that sample rate.
  - jeffbee 3y ago
    That's not true in all models. For example in the (execrable) `statsd` model there is a bit of information sent every time a metric changes.
    - deathanatos 3y ago
      It's not just the stats protocol, it's the underlying metric, too. statsd is just a way of recording/transmitting metrics. If I transmit a statsd metric representing "CPU usage", I am still sampling it. E.g., I might read the CPU usage every second & generate a statsd stat. That's a sample rate of 1Hz. I have to choose some sampling frequency, since the API most OS's expose is "what's the current CPU usage?". If the metric is "total number of HTTP requests", then I can definitely just transmit that metric every time I get a request. We're not sampling for that metric. The latter is inherently a discrete event, with which we can know every data point of, though. Things like CPU, memory, are either fundamentally continuous, or their implementations are simply sampling it. I do agree the model matters too; Prom's tendency to just poll /metrics endpoints every n seconds means even things like HTTP events are inherently sampled.
      - ekimekim 3y ago
        > If I transmit a statsd metric representing "CPU usage", I am still sampling it. In practice this is how everyone does it, but in theory it should be possible to have a non-sampled view of CPU usage (defined as "time process is scheduled onto a CPU"). With the right kernel introspection, you could represent it as a series of spans covering each time slice where the process is scheduled. Perhaps with a concept of a "currently ongoing span" to account for the current time slice. Do I think this would be more useful than the typical sampled metric? Probably not, outside some niche performance analysis workflows. But my point is that CPU is not actually continuous, and I struggle to think of any metric which cannot be represented without sampling if you REALLY need it.