4 ms·
Still waiting for someone to open source an observability db that uses distributions as the core storage abstraction instead of just single values. I thought C
by kasey_junk 6y ago
Still waiting for someone to open source an observability db that uses distributions as the core storage abstraction instead of just single values.
I thought Circonus’ IronDB was open source but they’ve either brought it back closed source or I was mistaken.
In any case it seems weird that everyone is just rehashing the same space with metrics dbs and not moving to where Google/Circonus have been for a while.
- ekabod 6y agoclickhouse as aggregate functions (Pearson correlation coefficient,quantiles , etc... )[0]. Associated with their Materialized view and Liveview features, you can achieve observability [1]. [0]https://clickhouse-docs.readthedocs.io/en/latest https://clickhouse-docs.readthedocs.io/en/latest [1]https://www.altinity.com/blog/2019/11/13/making-data-come-to-life-with-clickhouse-live-view-tables https://www.altinity.com/blog/2019/11/13/making-data-come-to... Edit: fixed broken link.
- alexhutcheson 6y agoVeneur[1] can compute approximate global percentiles (among other things) during metric collection, and the percentiles can be stored and queried in even in a datastore that doesn't know anything about distributions. [1] https://github.com/stripe/veneur https://github.com/stripe/veneur
- kasey_junk 6y agoBut not across dimensionality and because veneur has outsourced the storage of the metrics you can’t calculate arbitrary percentiles after the fact.
- zeeboo 6y agoI've been working on something like this for some years now: https://github.com/zeebo/rothko https://github.com/zeebo/rothko
- otterley 6y agoI'm not sure I understand. Google, Circonus, etc. all store distributions as scalar values. A histogram is merely a collection of scalar values, where the collection is a set of bounded buckets, where the boundaries are typically adjacent to one another (e.g., 0.1 <= x < 0.2, 0.2 <= x < 0.3, etc.). You can do this with any timeseries DB. The aggregation functionality is best done in a separate layer so that you have the flexibility to change how you create your distributions without having to change your storage backend.
- kasey_junk 6y agoThey store scalar values in that they are binary encoded data structures. But the data structures they actually encode at the cell levels they histogram bins in a datastructure optimized for histogram calculations (in the Circonus case log linear histograms encoded as 3 values[0]). These histogram structures are optimized for fast re-aggregation and quantization across time/dimension. By optimizing for histograms first and plugging things like counts and gauges into them (rather than the reverse for something like Prometheus) they get around a lot of the bad statistics and expensive query calculations that other metrics systems have. This is different than say how Prometheus stores histograms where it treats the bucket boundaries as a dimension in a fixed bucket histogram and the cell as the count of items in that bucket. [0] https://www.circonus.com/2018/05/effective-management-of-high-volume-numeric-data-with-histograms/ https://www.circonus.com/2018/05/effective-management-of-hig...
- otterley 6y agoThanks for the link! This is some good reading.