4 ms·

I love prometheus. Its simple and the built in charts are enough without having to use Grafana on top.

halfmatthalfcat 6y ago
The question is how to do long term storage though. Something I've had a bit of trouble rationing about. Right now all of my metrics are sitting in a PVC with a 30d retention period, so we're probably fine but for longer term cold storage the options aren't great unless you want to run a custom Postgres instance with the Timescale plugin or something else more managed.
- x87678r 6y ago
  Do you really need long term? I hate throwing away data but realistically I never really need old performance data. Some stats data is worth keeping but you can extract a few important time series and store them elsewhere.
  - julioo 6y ago
    In any case, Prometheus is throwing away data if a scrap can’t be done. As clearly described on the website, prometheus is not a metrics system. So influx and Prometheus are quite different.
    - julioo 6y ago
      Mistake in my previous message. I wanted to say that Prometheus is not a log system. Metrics could be lost in scraping issues. This is ok in some cases but you have to know that you can loose data.
- dewey 6y ago
  I think that's what https://thanos.io/ https://thanos.io/ is for.
  - halfmatthalfcat 6y ago
    Thanks for the link, I'll check it out.
- sevagh 6y ago
  I second the Thanos link, but if you want to only use Prometheus, I would look into federation: https://www.robustperception.io/scaling-and-federating-prometheus https://www.robustperception.io/scaling-and-federating-prome... https://prometheus.io/docs/prometheus/latest/federation/ https://prometheus.io/docs/prometheus/latest/federation/ You can set up a new "cold storage" Prometheus with a longer retention that scrapes select metrics from your regular 30d Prometheus to store for longer periods of time.
- dfinninger 6y ago
  How about another Prometheus server in federation[0]? It's as simple as another Prom instance pointing at the "live" Prom instance. You can filter only the metrics you want to keep for a long amount of time, and downsample if necessary (by just setting a higher scrape interval on the "long term" Prom). Since this "long term" Prom isn't in the critical path you could skimp on processing resources and just give it a big disk as a cost optimization. If you are on AWS (or equivalent) the storage there is pretty durable. On-prem you can run HA (two instances with identical config). [0] https://prometheus.io/docs/prometheus/latest/federation/ https://prometheus.io/docs/prometheus/latest/federation/
- codeduck 6y ago
  For long term retention, look at VictoriaMetrics
  - hagen1778 6y ago
    My vote is for VictoriaMetrics - the best option if you want to keep things simple and get great performance in the same time!
    - mekster 6y ago
      It's sad to see technically inferior products having more popularity. VictoriaMetrics pretty much has everything you'd hope for as a Prometheus long term storage, like direct PromQL support, good performance and ease of installation.
- deepakhj 6y ago
  Thanos or Cortex
speedgoose 6y ago
I don't. It's quite difficult to do complex queries and its query language has a few gotcha. I think it's quite good and one of the best solutions today, but I look forward to something as simple and as fast, but with a proper query language.
- x87678r 6y ago
  Yes would be nice to be able to run sql or something more standard. I like that its just simple, one application and it just works.
- valyala 6y ago
  PromQL from Prometheus has steep learning curve because of lack of good documentation. When you understand PromQL basics, then it is much easier to write typical queries over time series data in PromQL than in any other query languages (SQL, Flux, InfluxQL, etc.). I'd recommend reading the article about PromQL basics [1] and then feel the power of PromQL. [1] https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085 https://medium.com/@valyala/promql-tutorial-for-beginners-9a...

The question is how to do long term storage though. Something I've had a bit of trouble rationing about. Right now all of my metrics are sitting in a PVC with a 30d retention period, so we're probably fine but for longer term cold storage the options aren't great unless you want to run a custom Postgres instance with the Timescale plugin or something else more managed.

x87678r 6y ago
Do you really need long term? I hate throwing away data but realistically I never really need old performance data. Some stats data is worth keeping but you can extract a few important time series and store them elsewhere.
- julioo 6y ago
  In any case, Prometheus is throwing away data if a scrap can’t be done. As clearly described on the website, prometheus is not a metrics system. So influx and Prometheus are quite different.
  - julioo 6y ago
    Mistake in my previous message. I wanted to say that Prometheus is not a log system. Metrics could be lost in scraping issues. This is ok in some cases but you have to know that you can loose data.
dewey 6y ago
I think that's what https://thanos.io/ https://thanos.io/ is for.
- halfmatthalfcat 6y ago
  Thanks for the link, I'll check it out.
sevagh 6y ago
I second the Thanos link, but if you want to only use Prometheus, I would look into federation: https://www.robustperception.io/scaling-and-federating-prometheus https://www.robustperception.io/scaling-and-federating-prome... https://prometheus.io/docs/prometheus/latest/federation/ https://prometheus.io/docs/prometheus/latest/federation/ You can set up a new "cold storage" Prometheus with a longer retention that scrapes select metrics from your regular 30d Prometheus to store for longer periods of time.
dfinninger 6y ago
How about another Prometheus server in federation[0]? It's as simple as another Prom instance pointing at the "live" Prom instance. You can filter only the metrics you want to keep for a long amount of time, and downsample if necessary (by just setting a higher scrape interval on the "long term" Prom). Since this "long term" Prom isn't in the critical path you could skimp on processing resources and just give it a big disk as a cost optimization. If you are on AWS (or equivalent) the storage there is pretty durable. On-prem you can run HA (two instances with identical config). [0] https://prometheus.io/docs/prometheus/latest/federation/ https://prometheus.io/docs/prometheus/latest/federation/
codeduck 6y ago
For long term retention, look at VictoriaMetrics
- hagen1778 6y ago
  My vote is for VictoriaMetrics - the best option if you want to keep things simple and get great performance in the same time!
  - mekster 6y ago
    It's sad to see technically inferior products having more popularity. VictoriaMetrics pretty much has everything you'd hope for as a Prometheus long term storage, like direct PromQL support, good performance and ease of installation.
deepakhj 6y ago
Thanos or Cortex

speedgoose 6y ago

I don't. It's quite difficult to do complex queries and its query language has a few gotcha. I think it's quite good and one of the best solutions today, but I look forward to something as simple and as fast, but with a proper query language.

x87678r 6y ago
Yes would be nice to be able to run sql or something more standard. I like that its just simple, one application and it just works.
valyala 6y ago
PromQL from Prometheus has steep learning curve because of lack of good documentation. When you understand PromQL basics, then it is much easier to write typical queries over time series data in PromQL than in any other query languages (SQL, Flux, InfluxQL, etc.). I'd recommend reading the article about PromQL basics [1] and then feel the power of PromQL. [1] https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085 https://medium.com/@valyala/promql-tutorial-for-beginners-9a...