4 ms·

Missing from the title: leaving InfluxDB and Prometheus for VictoriaMetrics.

hintymad 3y ago
This is puzzling. I'm not sure how VictoriaMetrics solved the cardinality problem? When running an aggregate query that sums up some counters for a single metric over the dimension of instances in a time window of larger than a few hours, VictoriaMetrics would barf with error for the querying having too many time series (or data points? I forgot the exact wording). This clearly shows that 1/ Victoria Metrics does not treat a time series with multiple dimensions as a single time series; 2/ VictoriaMetrics does not perform hierarchical aggregation. That is, VictoriaMetrics has not really built a true time series DB that handles reasonable cardinalities.
- deleted 3y ago
  [deleted]
- narism 3y ago
  There are a number of circuit breakers in VictoriaMetrics that limit the number of time series/datapoints in queries to limit CPU/RAM usage. These can be tweaked with the -search.max* command line flags. I think the default query limits for data points per series are 30M and number of series is 300K. What do you consider reasonable cardinalities or a true TSDB?
  - hintymad 3y ago
    In my particular example, the cardinality of the instances (i.e., the number of of unit instance count in a query's time range) should not even matter. I was summing a counter over all the instances, and VictoriaMetrics should just add up the counters for every time unit while scanning all the data points -- this is implemented by pretty much all the OLAP engines. Or put it another way, logically I was query over a single time series. It's just that each value in the time series was associated with multiple values. Given such logical model, a good time series database should not not even bother me with any concern of cardinality.
- dikei 3y ago
  https://github.com/VictoriaMetrics/VictoriaMetrics#cardinality-limiter https://github.com/VictoriaMetrics/VictoriaMetrics#cardinali... If I understand this correctly, it deals with high cardinality by dropping data. The operators need to monitor for this and adjust their data to lower the cardinality.
  - narism 3y ago
    From your link: > By default VictoriaMetrics doesn't limit the number of stored time series. They have put out some benchmarks showing VictoriaMetrics ingesting 40M time series: https://valyala.medium.com/high-cardinality-tsdb-benchmarks-victoriametrics-vs-timescaledb-vs-influxdb-13e6ee64dd6b https://valyala.medium.com/high-cardinality-tsdb-benchmarks-...
  - eclark 3y ago
    This is the only longer term scalable solution. High cardinality for TSDB's have to be dealt with by dropping. Or you run out of storage, write rate, memory, or network. It's possible to smooth this loss out (by assuming a normal distribution of lost data) if it's noticed and limited. Though I do not think there's any commercial TSDB that does that automatically.

This is puzzling. I'm not sure how VictoriaMetrics solved the cardinality problem? When running an aggregate query that sums up some counters for a single metric over the dimension of instances in a time window of larger than a few hours, VictoriaMetrics would barf with error for the querying having too many time series (or data points? I forgot the exact wording). This clearly shows that 1/ Victoria Metrics does not treat a time series with multiple dimensions as a single time series; 2/ VictoriaMetrics does not perform hierarchical aggregation. That is, VictoriaMetrics has not really built a true time series DB that handles reasonable cardinalities.

deleted 3y ago
[deleted]
narism 3y ago
There are a number of circuit breakers in VictoriaMetrics that limit the number of time series/datapoints in queries to limit CPU/RAM usage. These can be tweaked with the -search.max* command line flags. I think the default query limits for data points per series are 30M and number of series is 300K. What do you consider reasonable cardinalities or a true TSDB?
- hintymad 3y ago
  In my particular example, the cardinality of the instances (i.e., the number of of unit instance count in a query's time range) should not even matter. I was summing a counter over all the instances, and VictoriaMetrics should just add up the counters for every time unit while scanning all the data points -- this is implemented by pretty much all the OLAP engines. Or put it another way, logically I was query over a single time series. It's just that each value in the time series was associated with multiple values. Given such logical model, a good time series database should not not even bother me with any concern of cardinality.
dikei 3y ago
https://github.com/VictoriaMetrics/VictoriaMetrics#cardinality-limiter https://github.com/VictoriaMetrics/VictoriaMetrics#cardinali... If I understand this correctly, it deals with high cardinality by dropping data. The operators need to monitor for this and adjust their data to lower the cardinality.
- narism 3y ago
  From your link: > By default VictoriaMetrics doesn't limit the number of stored time series. They have put out some benchmarks showing VictoriaMetrics ingesting 40M time series: https://valyala.medium.com/high-cardinality-tsdb-benchmarks-victoriametrics-vs-timescaledb-vs-influxdb-13e6ee64dd6b https://valyala.medium.com/high-cardinality-tsdb-benchmarks-...
- eclark 3y ago
  This is the only longer term scalable solution. High cardinality for TSDB's have to be dealt with by dropping. Or you run out of storage, write rate, memory, or network. It's possible to smooth this loss out (by assuming a normal distribution of lost data) if it's noticed and limited. Though I do not think there's any commercial TSDB that does that automatically.