3 ms·

It's not just the stats protocol, it's the underlying metric, too. statsd is just a way of recording/transmitting metrics. If I transmit a statsd metric repres

It's not just the stats protocol, it's the underlying metric, too. statsd is just a way of recording/transmitting metrics. If I transmit a statsd metric representing "CPU usage", I am still sampling it. E.g., I might read the CPU usage every second & generate a statsd stat. That's a sample rate of 1Hz. I have to choose some sampling frequency, since the API most OS's expose is "what's the current CPU usage?". If the metric is "total number of HTTP requests", then I can definitely just transmit that metric every time I get a request. We're not sampling for that metric. The latter is inherently a discrete event, with which we can know every data point of, though. Things like CPU, memory, are either fundamentally continuous, or their implementations are simply sampling it. I do agree the model matters too; Prom's tendency to just poll /metrics endpoints every n seconds means even things like HTTP events are inherently sampled.

ekimekim 3y ago
> If I transmit a statsd metric representing "CPU usage", I am still sampling it. In practice this is how everyone does it, but in theory it should be possible to have a non-sampled view of CPU usage (defined as "time process is scheduled onto a CPU"). With the right kernel introspection, you could represent it as a series of spans covering each time slice where the process is scheduled. Perhaps with a concept of a "currently ongoing span" to account for the current time slice. Do I think this would be more useful than the typical sampled metric? Probably not, outside some niche performance analysis workflows. But my point is that CPU is not actually continuous, and I struggle to think of any metric which cannot be represented without sampling if you REALLY need it.
- deathanatos 3y ago
  I almost put exactly that in a footnote, 'cept about RAM, instead of CPU usage. No OS that I know of exposes such an API, so it's highly theoretical. As for truly contiguous metrics, hmm. How about current battery charge (in Wh)? Host uptime also seems technically continuous (albeit representable by a straight line). (Yes, we track this met; it makes reboots stand out in lieu of my metrics system not providing a vertical marker feature.) Clock drift? (and I'm going to insert the footnote on this comment about something something Planck units.)

ekimekim 3y ago

> If I transmit a statsd metric representing "CPU usage", I am still sampling it. In practice this is how everyone does it, but in theory it should be possible to have a non-sampled view of CPU usage (defined as "time process is scheduled onto a CPU"). With the right kernel introspection, you could represent it as a series of spans covering each time slice where the process is scheduled. Perhaps with a concept of a "currently ongoing span" to account for the current time slice. Do I think this would be more useful than the typical sampled metric? Probably not, outside some niche performance analysis workflows. But my point is that CPU is not actually continuous, and I struggle to think of any metric which cannot be represented without sampling if you REALLY need it.

deathanatos 3y ago
I almost put exactly that in a footnote, 'cept about RAM, instead of CPU usage. No OS that I know of exposes such an API, so it's highly theoretical. As for truly contiguous metrics, hmm. How about current battery charge (in Wh)? Host uptime also seems technically continuous (albeit representable by a straight line). (Yes, we track this met; it makes reboots stand out in lieu of my metrics system not providing a vertical marker feature.) Clock drift? (and I'm going to insert the footnote on this comment about something something Planck units.)