3 ms·

Did you start with Victoria directly? Or move to it from Prometheus? Keen to hear your experiences. We're currently running Prometheus + Thanos, and high cardi

Did you start with Victoria directly? Or move to it from Prometheus? Keen to hear your experiences. We're currently running Prometheus + Thanos, and high cardinality timeseries are a real issue, which Victoria claims to be good at.

harkishen 6y ago
High cardinality on Promscale is very efficient. I have been testing prom-migrator, migrating and ingesting 7 million+ samples a second, and till now, I have seen Promscale's memory usage to be the most efficient, along with a better CPU performance. I have tested Promscale, VictoriaMetrics, Thanos, M3, Cortex, and Promscale continues to lead in efficiency and performance. Also, the added advantage of native SQL support is a big bonus.
shitloadofbooks 6y ago
We went from Prometheus (bursting at the seams and begging to be scaled) -> a failed Cortex implementation -> a rock solid Victoria Metrics implementation. We found Cortex to be extremely over-engineered, extremely hard to tune (because of multiple configuration/argument refactors with incomplete doc cleanups) and found that it would just fall over under load. (Though to be fair, there wasn't much in the way of first-party deployment tooling at the time, so it was a hand-rolled Helm chart and at least 50% of the issues were likely my config). In comparison, the first-party Victoria Metrics Helm chart worked straight out of the box, the maintainers have fixed multiple small issues within hours of me reporting them and we've thrown an extremely large amount of metrics at it with zero problems.
cevian 6y ago
(Promscale Team Lead Here) Promscale, which connects Prometheus to TimescaleDB also handles high-cardinality well and allows you to use both PromQL and SQL for data analysis. I'd humbly suggest taking a look. Plus our PromQL implementation is 100% standards compliant.
- valyala 6y ago
  It would be great conducting a benchmark between Promscale and VictoriaMetrics similar to this one - https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f https://valyala.medium.com/prometheus-vs-victoriametrics-ben... . It differs from TSBS in the following aspects: * It is based on real-world data instead of synthetic data. * It measures resources usage (RAM, CPU, disk IO, disk space) during production-like load instead of measuring peak performance under the maximum synthetic load.
  - pointestimate 6y ago
    I would love to have some kind of integration between Victoria Metrics and Promscale, using VM for ingest from things like InfluxDB and Promscale for SQL queries on stable data.
    - valyala 6y ago
      This should be already possible - just collect Influx line protocol data with vmagent [1] and write it to Promscale. [1] https://victoriametrics.github.io/vmagent.html https://victoriametrics.github.io/vmagent.html

harkishen 6y ago

High cardinality on Promscale is very efficient. I have been testing prom-migrator, migrating and ingesting 7 million+ samples a second, and till now, I have seen Promscale's memory usage to be the most efficient, along with a better CPU performance. I have tested Promscale, VictoriaMetrics, Thanos, M3, Cortex, and Promscale continues to lead in efficiency and performance. Also, the added advantage of native SQL support is a big bonus.

shitloadofbooks 6y ago

We went from Prometheus (bursting at the seams and begging to be scaled) -> a failed Cortex implementation -> a rock solid Victoria Metrics implementation. We found Cortex to be extremely over-engineered, extremely hard to tune (because of multiple configuration/argument refactors with incomplete doc cleanups) and found that it would just fall over under load. (Though to be fair, there wasn't much in the way of first-party deployment tooling at the time, so it was a hand-rolled Helm chart and at least 50% of the issues were likely my config). In comparison, the first-party Victoria Metrics Helm chart worked straight out of the box, the maintainers have fixed multiple small issues within hours of me reporting them and we've thrown an extremely large amount of metrics at it with zero problems.

cevian 6y ago

(Promscale Team Lead Here) Promscale, which connects Prometheus to TimescaleDB also handles high-cardinality well and allows you to use both PromQL and SQL for data analysis. I'd humbly suggest taking a look. Plus our PromQL implementation is 100% standards compliant.

valyala 6y ago
It would be great conducting a benchmark between Promscale and VictoriaMetrics similar to this one - https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f https://valyala.medium.com/prometheus-vs-victoriametrics-ben... . It differs from TSBS in the following aspects: * It is based on real-world data instead of synthetic data. * It measures resources usage (RAM, CPU, disk IO, disk space) during production-like load instead of measuring peak performance under the maximum synthetic load.
- pointestimate 6y ago
  I would love to have some kind of integration between Victoria Metrics and Promscale, using VM for ingest from things like InfluxDB and Promscale for SQL queries on stable data.
  - valyala 6y ago
    This should be already possible - just collect Influx line protocol data with vmagent [1] and write it to Promscale. [1] https://victoriametrics.github.io/vmagent.html https://victoriametrics.github.io/vmagent.html