3 ms·

Is it really that different from, say, the way Prometheus supports histogram-based quantiles? https://prometheus.io/docs/practices/histograms/ https://prometheu

Is it really that different from, say, the way Prometheus supports histogram-based quantiles? https://prometheus.io/docs/practices/histograms/ https://prometheus.io/docs/practices/histograms/ Granted, it looks like Monarch supports a more cleanly-defined schema for distributions, whereas Prometheus just relies on you to define the buckets yourself and follow the convention of using a "le" label to expose them. But the underlying representation (an empirical CDF) seems to be the same, and so the accuracy tradeoffs should also be the same.

spullara 4y ago
Much different. When you are reporting histograms you can combine them and see the true p50 or whatever across all the individual systems reporting the metric.
- teraflop 4y ago
  That is also possible in Prometheus, which is why I made the comparison.
- deleted 4y ago
  [deleted]
- gttalbot 4y ago
  Yes. This. Also, displaying histograms in heatmap format can allow you to intuit the behavior of layered distributed systems, caches, etc. Relatedly, exemplars allowed tying related data to histogram buckets. For example, RPC traces could be tied to the latency bucket & time at which they complete, giving a natural means to tie metrics monitoring and tracing, so you can "go to the trace with the problem". This is described in the paper as well.
- nvarsj 4y ago
  Can you elaborate a bit? You can do the same in Prometheus by summing the bucket counts. Not sure what you mean by “true p50” either. With buckets it’s always an approximation based on the bucket widths.
  - spullara 4y ago
    Ah, I misunderstood what you meant. If you are reporting static buckets I get how that is better than what folks typically do but how do you know the buckets a priori? Others back their histograms with things like https://github.com/tdunning/t-digest https://github.com/tdunning/t-digest. It is pretty powerful as the buckets are dynamic based on the data and histograms can be added together.

spullara 4y ago

Much different. When you are reporting histograms you can combine them and see the true p50 or whatever across all the individual systems reporting the metric.

teraflop 4y ago
That is also possible in Prometheus, which is why I made the comparison.
deleted 4y ago
[deleted]
gttalbot 4y ago
Yes. This. Also, displaying histograms in heatmap format can allow you to intuit the behavior of layered distributed systems, caches, etc. Relatedly, exemplars allowed tying related data to histogram buckets. For example, RPC traces could be tied to the latency bucket & time at which they complete, giving a natural means to tie metrics monitoring and tracing, so you can "go to the trace with the problem". This is described in the paper as well.
nvarsj 4y ago
Can you elaborate a bit? You can do the same in Prometheus by summing the bucket counts. Not sure what you mean by “true p50” either. With buckets it’s always an approximation based on the bucket widths.
- spullara 4y ago
  Ah, I misunderstood what you meant. If you are reporting static buckets I get how that is better than what folks typically do but how do you know the buckets a priori? Others back their histograms with things like https://github.com/tdunning/t-digest https://github.com/tdunning/t-digest. It is pretty powerful as the buckets are dynamic based on the data and histograms can be added together.