5 ms·

In my experience, Monarch storing histograms and being unable to rebucket on the fly is a big problem. A percentile line on a histogram will be incredibly misle

In my experience, Monarch storing histograms and being unable to rebucket on the fly is a big problem. A percentile line on a histogram will be incredibly misleading, because it's trying to figure out what the p50 of a bunch of buckets is. You'll see monitoring artifacts like large jumps and artificial plateaus as a result of how requests fall into buckets. The bucketer on the default RPC latency metric might not be well tuned for your service. I've seen countless experienced oncallers tripped up by this, because "my graphs are lying to me" is not their first thought.

jrockway 4y ago
I definitely remember a lot of time spent tweaking histogram buckets for performance vs. accuracy. The default bucketing algorithm at the time was powers of 4 or something very unusual like that.
- shadowgovt 4y ago
  It's because powers of four was great for the original application of statistics on high traffic services where the primary thing the user was interested in was deviations from the norm, and with a high traffic system the signal for what the norm is would be very strong. I tried applying it to a service with much lower traffic and found the bucketing to be extremely fussy.
deleted 4y ago
[deleted]
heinrichhartman 4y ago
Circonus Histograms solve that by using a universal bucketing scheme. Details are explained in this paper: https://arxiv.org/abs/2001.06561 https://arxiv.org/abs/2001.06561 Disclaimer: I am a co-author.
- mherdeg 4y ago
  Wow, this is a fantastic solution to some questions I've had rattling around in my head for years about the optimal bucket choices to minimize error given a particular set of buckets. Do I read right that circllhist has a pretty big number of bin sizes and is not configurable (except that they're sparse so may be small on disk)? I've found myself using high-cardinality Prometheus metrics where I can only afford 10-15 distinct histogram buckets. So I end up (1) plugging in my live system data from normal operations and from outage periods into various numeric algorithms that propose optimal bucket boundaries. These algorithms tell me that I could get great accuracy if I chose thousands of buckets, which, thanks for rubbing it in about my space problems :(. Then I write some more code to collapse those into 15 buckets while minimizing error at various places (like p50, p95, p99, p999 under normal operations and under irregular operations). (2) making sure I have an explicit bucket boundary at any target that represents a business objective (if my service promises no more than 1% of requests will take >2500ms, setting a bucket boundary at 2500ms gives me perfectly precise info about whether p99 falls above/below 2500ms) (3) forgetting to tune this and leaving a bunch of bad defaults in place which often lead to people saying "well, our graph shows a big spike up to 10000ms but that's just because we forgot to tune our histogram bucket boundaries before the outage, actually we have to refer to logs to see the timeouts at 50 sec"
  - joeblubaugh 4y ago
    I’ve used these log-linear history in a few pieces of code. There is some configurability in the abstract - you could choose a different logarithm base. In practice none of the implementations seem to provide that. Within the each set of buckets for a given log base you have reasonable precision at that magnitude. If your metric is oscillating around 1e6 you shouldn’t care much about the variance at 1e2, and with this scheme you don’t have to tune anything to provide for that.
    - heinrichhartman 4y ago
      There are a large amount of subtle tradeoffs around the bucketing scheme (log, vs. log-linear, base) and memory layout (sparse, dense, chunked) the amount of configurability in the histogram space (circllhist, DDSketch, HDRHistogram, ...). A good overview is this discussion here: https://github.com/open-telemetry/opentelemetry-specification/issues/1776 https://github.com/open-telemetry/opentelemetry-specificatio... As for the circllhist: There are no knobs to turn. It uses base 10 and two decimal digits of precision. In the last 8 years I have not seen a single use-case in the operational domain where this was not appropriate.
- NHQ 4y ago
  Is it lossy to store data this way?
  - heinrichhartman 4y ago
    Yes. But also MUCH cheaper. You loose (1) ordering (2) ~5% of precision of the individual measurements. In practice, this still allows for highly precise percentile calculations (<0.1% error), see the evaluation in the Circllhist paper.
- dan-robertson 4y ago
  You’ve a typo in def 3.20. Should be ‘losing’. And another in §3.4, should be ‘continuous’. I didn’t thoroughly look for more.
kasey_junk 4y ago
My personal opinion is that they should have done a log linear histogram which solves the problems you mention (with other trade offs) but to me the big news was making the db flexible enough to have that data type. Leaving the world of single numeric type for each datum will influence the next generation of open source metrics db.
- gttalbot 4y ago
  Yeah in theory people could do their own custom bucketing functions. Would be worth researching log-linear for that certainly.
gttalbot 4y ago
Yeah it was a tough tradeoff for the default case, because the team didn't want to use too much memory in everyone's binary since the RPC metrics were on by default. This is easily changeable by the user if necessary, though.
fleventynine 4y ago
The histograms are useful on their own (visualized as a heatmap). If percentile lines are necessary (and they often aren't), I prefer to overlay them on top of the heatmap so it is clear where the bucket edges are.

jrockway 4y ago

I definitely remember a lot of time spent tweaking histogram buckets for performance vs. accuracy. The default bucketing algorithm at the time was powers of 4 or something very unusual like that.

shadowgovt 4y ago
It's because powers of four was great for the original application of statistics on high traffic services where the primary thing the user was interested in was deviations from the norm, and with a high traffic system the signal for what the norm is would be very strong. I tried applying it to a service with much lower traffic and found the bucketing to be extremely fussy.

deleted 4y ago

[deleted]

heinrichhartman 4y ago

Circonus Histograms solve that by using a universal bucketing scheme. Details are explained in this paper: https://arxiv.org/abs/2001.06561 https://arxiv.org/abs/2001.06561 Disclaimer: I am a co-author.

mherdeg 4y ago
Wow, this is a fantastic solution to some questions I've had rattling around in my head for years about the optimal bucket choices to minimize error given a particular set of buckets. Do I read right that circllhist has a pretty big number of bin sizes and is not configurable (except that they're sparse so may be small on disk)? I've found myself using high-cardinality Prometheus metrics where I can only afford 10-15 distinct histogram buckets. So I end up (1) plugging in my live system data from normal operations and from outage periods into various numeric algorithms that propose optimal bucket boundaries. These algorithms tell me that I could get great accuracy if I chose thousands of buckets, which, thanks for rubbing it in about my space problems :(. Then I write some more code to collapse those into 15 buckets while minimizing error at various places (like p50, p95, p99, p999 under normal operations and under irregular operations). (2) making sure I have an explicit bucket boundary at any target that represents a business objective (if my service promises no more than 1% of requests will take >2500ms, setting a bucket boundary at 2500ms gives me perfectly precise info about whether p99 falls above/below 2500ms) (3) forgetting to tune this and leaving a bunch of bad defaults in place which often lead to people saying "well, our graph shows a big spike up to 10000ms but that's just because we forgot to tune our histogram bucket boundaries before the outage, actually we have to refer to logs to see the timeouts at 50 sec"
- joeblubaugh 4y ago
  I’ve used these log-linear history in a few pieces of code. There is some configurability in the abstract - you could choose a different logarithm base. In practice none of the implementations seem to provide that. Within the each set of buckets for a given log base you have reasonable precision at that magnitude. If your metric is oscillating around 1e6 you shouldn’t care much about the variance at 1e2, and with this scheme you don’t have to tune anything to provide for that.
  - heinrichhartman 4y ago
    There are a large amount of subtle tradeoffs around the bucketing scheme (log, vs. log-linear, base) and memory layout (sparse, dense, chunked) the amount of configurability in the histogram space (circllhist, DDSketch, HDRHistogram, ...). A good overview is this discussion here: https://github.com/open-telemetry/opentelemetry-specification/issues/1776 https://github.com/open-telemetry/opentelemetry-specificatio... As for the circllhist: There are no knobs to turn. It uses base 10 and two decimal digits of precision. In the last 8 years I have not seen a single use-case in the operational domain where this was not appropriate.
NHQ 4y ago
Is it lossy to store data this way?
- heinrichhartman 4y ago
  Yes. But also MUCH cheaper. You loose (1) ordering (2) ~5% of precision of the individual measurements. In practice, this still allows for highly precise percentile calculations (<0.1% error), see the evaluation in the Circllhist paper.
dan-robertson 4y ago
You’ve a typo in def 3.20. Should be ‘losing’. And another in §3.4, should be ‘continuous’. I didn’t thoroughly look for more.

kasey_junk 4y ago

My personal opinion is that they should have done a log linear histogram which solves the problems you mention (with other trade offs) but to me the big news was making the db flexible enough to have that data type. Leaving the world of single numeric type for each datum will influence the next generation of open source metrics db.

gttalbot 4y ago
Yeah in theory people could do their own custom bucketing functions. Would be worth researching log-linear for that certainly.

gttalbot 4y ago

Yeah it was a tough tradeoff for the default case, because the team didn't want to use too much memory in everyone's binary since the RPC metrics were on by default. This is easily changeable by the user if necessary, though.

fleventynine 4y ago

The histograms are useful on their own (visualized as a heatmap). If percentile lines are necessary (and they often aren't), I prefer to overlay them on top of the heatmap so it is clear where the bucket edges are.