5 ms·

The question is how to do long term storage though. Something I've had a bit of trouble rationing about. Right now all of my metrics are sitting in a PVC with

The question is how to do long term storage though. Something I've had a bit of trouble rationing about. Right now all of my metrics are sitting in a PVC with a 30d retention period, so we're probably fine but for longer term cold storage the options aren't great unless you want to run a custom Postgres instance with the Timescale plugin or something else more managed.

x87678r 6y ago
Do you really need long term? I hate throwing away data but realistically I never really need old performance data. Some stats data is worth keeping but you can extract a few important time series and store them elsewhere.
- julioo 6y ago
  In any case, Prometheus is throwing away data if a scrap can’t be done. As clearly described on the website, prometheus is not a metrics system. So influx and Prometheus are quite different.
  - julioo 6y ago
    Mistake in my previous message. I wanted to say that Prometheus is not a log system. Metrics could be lost in scraping issues. This is ok in some cases but you have to know that you can loose data.
dewey 6y ago
I think that's what https://thanos.io/ https://thanos.io/ is for.
- halfmatthalfcat 6y ago
  Thanks for the link, I'll check it out.
sevagh 6y ago
I second the Thanos link, but if you want to only use Prometheus, I would look into federation: https://www.robustperception.io/scaling-and-federating-prometheus https://www.robustperception.io/scaling-and-federating-prome... https://prometheus.io/docs/prometheus/latest/federation/ https://prometheus.io/docs/prometheus/latest/federation/ You can set up a new "cold storage" Prometheus with a longer retention that scrapes select metrics from your regular 30d Prometheus to store for longer periods of time.
dfinninger 6y ago
How about another Prometheus server in federation[0]? It's as simple as another Prom instance pointing at the "live" Prom instance. You can filter only the metrics you want to keep for a long amount of time, and downsample if necessary (by just setting a higher scrape interval on the "long term" Prom). Since this "long term" Prom isn't in the critical path you could skimp on processing resources and just give it a big disk as a cost optimization. If you are on AWS (or equivalent) the storage there is pretty durable. On-prem you can run HA (two instances with identical config). [0] https://prometheus.io/docs/prometheus/latest/federation/ https://prometheus.io/docs/prometheus/latest/federation/
codeduck 6y ago
For long term retention, look at VictoriaMetrics
- hagen1778 6y ago
  My vote is for VictoriaMetrics - the best option if you want to keep things simple and get great performance in the same time!
  - mekster 6y ago
    It's sad to see technically inferior products having more popularity. VictoriaMetrics pretty much has everything you'd hope for as a Prometheus long term storage, like direct PromQL support, good performance and ease of installation.
deepakhj 6y ago
Thanos or Cortex

x87678r 6y ago

Do you really need long term? I hate throwing away data but realistically I never really need old performance data. Some stats data is worth keeping but you can extract a few important time series and store them elsewhere.

julioo 6y ago
In any case, Prometheus is throwing away data if a scrap can’t be done. As clearly described on the website, prometheus is not a metrics system. So influx and Prometheus are quite different.
- julioo 6y ago
  Mistake in my previous message. I wanted to say that Prometheus is not a log system. Metrics could be lost in scraping issues. This is ok in some cases but you have to know that you can loose data.

dewey 6y ago

I think that's what https://thanos.io/ https://thanos.io/ is for.

halfmatthalfcat 6y ago
Thanks for the link, I'll check it out.

sevagh 6y ago

I second the Thanos link, but if you want to only use Prometheus, I would look into federation: https://www.robustperception.io/scaling-and-federating-prometheus https://www.robustperception.io/scaling-and-federating-prome... https://prometheus.io/docs/prometheus/latest/federation/ https://prometheus.io/docs/prometheus/latest/federation/ You can set up a new "cold storage" Prometheus with a longer retention that scrapes select metrics from your regular 30d Prometheus to store for longer periods of time.

dfinninger 6y ago

How about another Prometheus server in federation[0]? It's as simple as another Prom instance pointing at the "live" Prom instance. You can filter only the metrics you want to keep for a long amount of time, and downsample if necessary (by just setting a higher scrape interval on the "long term" Prom). Since this "long term" Prom isn't in the critical path you could skimp on processing resources and just give it a big disk as a cost optimization. If you are on AWS (or equivalent) the storage there is pretty durable. On-prem you can run HA (two instances with identical config). [0] https://prometheus.io/docs/prometheus/latest/federation/ https://prometheus.io/docs/prometheus/latest/federation/

codeduck 6y ago

For long term retention, look at VictoriaMetrics

hagen1778 6y ago
My vote is for VictoriaMetrics - the best option if you want to keep things simple and get great performance in the same time!
- mekster 6y ago
  It's sad to see technically inferior products having more popularity. VictoriaMetrics pretty much has everything you'd hope for as a Prometheus long term storage, like direct PromQL support, good performance and ease of installation.

deepakhj 6y ago

Thanos or Cortex