5 ms·

If you get serious about Prometheus, eventually you will want longer data retention, checkout https://thanos.io/ https://thanos.io/ Also sometimes you'll have

If you get serious about Prometheus, eventually you will want longer data retention, checkout https://thanos.io/ https://thanos.io/ Also sometimes you'll have some sort of a batch job, or a cron that can't be scrapped the regular way, checkout Prometheus Pushgateway https://prometheus.io/docs/practices/pushing/ https://prometheus.io/docs/practices/pushing/

ilyt 4y ago
I really don't get why "scrape the prometheus endpoint" is a go-to now, push model seems to be way less PITA to manage at scale. > If you get serious about Prometheus, eventually you will want longer data retention, checkout https://thanos.io/ https://thanos.io/ Any idea how it compares with https://victoriametrics.com/ https://victoriametrics.com/ ? We're slowly looking for a replacement for InfluxDB (as 1.8 is essentially on life support), the low disk footprint is pretty big advantage here.
- Raed667 4y ago
  > push model seems to be way less PITA to manage They have a blog post explaining their reasoning: https://prometheus.io/blog/2016/07/23/pull-does-not-scale-or-does-it/ https://prometheus.io/blog/2016/07/23/pull-does-not-scale-or...
- buro9 4y ago
  Scrape is typically just how people get started and works well for small and medium things, it gets you a long way before you need to consider it. Prometheus remote_write is what people graduate to, this gets you the rest of the way, and you are correct it's less PITA at scale. If you're looking for retention your choices are large, there's Cortex (CNCF), Mimir (most Cortex work moved here), Thanos, VictoriaMetrics, TimeScale, Chronosphere, and many others. All seek to do a similar thing from a distance, they all store metrics (likely from Prometheus) and allow retention and some variety of how to query it (if you want SQL you got it, if you want non-standard functions you go it, if your reads are more important than your writes you got it, if you need a billion active series you got it, etc). If what you want is "Prometheus but bigger" then the Prometheus project provides a compliance suite that you can run to help you evaluate your options: https://github.com/prometheus/compliance https://github.com/prometheus/compliance I work for Grafana Labs, and we have maintainers working for us who have touched Prometheus, Thanos, Cortex and Mimir. Mimir is currently the largest investment we have https://github.com/grafana/mimir https://github.com/grafana/mimir and it is 100% compliant with Prometheus (though that is about to be temporarily untrue as Native Histograms is landing in Prometheus soon https://github.com/prometheus/prometheus/milestone/10 https://github.com/prometheus/prometheus/milestone/10 and we'll need to add a perfectly compliant support to Mimir to get back to being compliant).
  - Rapzid 4y ago
    Interesting... I recall the prom creators being very, very convinced that pull was the one true way (push gateway was viewed as a compatibility concession).
    - capableweb 4y ago
      I mean, in most cases it just works, for typical "Here is my HTTP server" services. But for batch processes, not so well, as you're not exposing a HTTP endpoint. Pull does work like 90% of the time, + you get a handy uptime check built-in as well, as if the pull didn't work, the service is probably offline and you'll notice it immediately. With Push you still need a way of checking if the service is online/offline.
- heliodor 4y ago
  VictoriaMetrics is super simple to use and reliable. It's a breath of fresh air compared to running other tools.
- oofnik 4y ago
  We began a transition from a single behemoth Prometheus instance to Victoria Metrics about a month ago after evaluating a couple of similar tools. VM won out because of its relative operational simplicity and eye-popping benchmark results in resource usage comparisons. Documentation can be a little rough around the edges, but overall we're impressed so far.
marginalia_nu 4y ago
Batch jobs are arguably a lot better served with old-school logging than metrics. If you're using prometheus for alerts, I'll maybe go as far as considering incrementing a "the batch job failed"-counter a good idea. But all this is very much jamming a square peg into a round hole.
- encoderer 4y ago
  Also, there are tools like Cronitor that can help.
cduzz 4y ago
"Hey, I'm trying to figure out some business metrics related to our infrastructure and application performance / uptime; do we have the data anywhere?" "Sure, Bob, just write a promql query against this interface" https://news.ycombinator.com/item?id=1636348 https://news.ycombinator.com/item?id=1636348 (in case my point is too subtle: I believe that the value of long term metric retention is to support business decisions more than supporting availability related monitoring. The tools "business deciders" use to look at "the things they call metrics" are simply unable to look at data lodged in a prometheus silo. So the data you have for long term retention is simply unusable for the thing it's most valuable) (edited for clarity)
- flerchin 4y ago
  I use thanos for availability related concerns. eg, Some metric crossed the threshold for saturation, and I can go back as far as I need to see when the metric began saturating. We have seasonal business patterns that I can show the high water-mark for resource consumption. Thanos also allows a CQRS-type pattern if that's important to folks. Do I point the biz at it? Only when I'm trying to show them availability related metrics.
  - deleted 4y ago
    [deleted]
- gjulianm 4y ago
  What tools do they use? It's not hard to build a dashboard in Grafana and show them there the metrics they want, in an easily-digestible format.
  - cduzz 4y ago
    There's an entire universe of people making business decisions who live in excel and "BI" tools. If your thing doesn't speak ODBC / JDBC it's invisible. Further, the vast majority of workaday devs are stressed out and under tremendous pressure to thing all the things in their ticket queue. They're probably pretty fluent in their thinging language and likely also conversant in SQL. Asking them "hey learn this other query language, because" is just adding stress to them. Sure, some are going to rise above and promql things too, but for lots, it's telling them to "write a distributed map reduce query in erlang."
packetslave 4y ago
I would highly recommend looking at Cortex (or Grafana Labs' fork, Mimir) instead of Thanos. It doesn't rely on reading the Prometheus WAL (rather, it uses Prometheus's built-in remote-write functionality). This is both less fragile and has the benefit of being about to run Prometheus in stateless mode (or replace it entirely with something like Grafana Agent, vmagent, etc. Grafana Labs has done a proof of concept showing Mimir scaling to 1 billion active timeseries. There are also "real" users serving 700+ million metrics in production.
- saifulwebid 4y ago
  How do you “import” existing Prometheus metrics block into Cortex/Mimir using the remote-write, then?
  - packetslave 4y ago
    For Thanos, you can (mostly) just copy the blocks from the existing s3 bucket into Mimir's bucket. There's some cleanup prep around external labels, but nothing too difficult, and the process is well-documented. I just finished migrating a year of Thanos metrics into Mimir without issue.

ilyt 4y ago

I really don't get why "scrape the prometheus endpoint" is a go-to now, push model seems to be way less PITA to manage at scale. > If you get serious about Prometheus, eventually you will want longer data retention, checkout https://thanos.io/ https://thanos.io/ Any idea how it compares with https://victoriametrics.com/ https://victoriametrics.com/ ? We're slowly looking for a replacement for InfluxDB (as 1.8 is essentially on life support), the low disk footprint is pretty big advantage here.

Raed667 4y ago
> push model seems to be way less PITA to manage They have a blog post explaining their reasoning: https://prometheus.io/blog/2016/07/23/pull-does-not-scale-or-does-it/ https://prometheus.io/blog/2016/07/23/pull-does-not-scale-or...
buro9 4y ago
Scrape is typically just how people get started and works well for small and medium things, it gets you a long way before you need to consider it. Prometheus remote_write is what people graduate to, this gets you the rest of the way, and you are correct it's less PITA at scale. If you're looking for retention your choices are large, there's Cortex (CNCF), Mimir (most Cortex work moved here), Thanos, VictoriaMetrics, TimeScale, Chronosphere, and many others. All seek to do a similar thing from a distance, they all store metrics (likely from Prometheus) and allow retention and some variety of how to query it (if you want SQL you got it, if you want non-standard functions you go it, if your reads are more important than your writes you got it, if you need a billion active series you got it, etc). If what you want is "Prometheus but bigger" then the Prometheus project provides a compliance suite that you can run to help you evaluate your options: https://github.com/prometheus/compliance https://github.com/prometheus/compliance I work for Grafana Labs, and we have maintainers working for us who have touched Prometheus, Thanos, Cortex and Mimir. Mimir is currently the largest investment we have https://github.com/grafana/mimir https://github.com/grafana/mimir and it is 100% compliant with Prometheus (though that is about to be temporarily untrue as Native Histograms is landing in Prometheus soon https://github.com/prometheus/prometheus/milestone/10 https://github.com/prometheus/prometheus/milestone/10 and we'll need to add a perfectly compliant support to Mimir to get back to being compliant).
- Rapzid 4y ago
  Interesting... I recall the prom creators being very, very convinced that pull was the one true way (push gateway was viewed as a compatibility concession).
  - capableweb 4y ago
    I mean, in most cases it just works, for typical "Here is my HTTP server" services. But for batch processes, not so well, as you're not exposing a HTTP endpoint. Pull does work like 90% of the time, + you get a handy uptime check built-in as well, as if the pull didn't work, the service is probably offline and you'll notice it immediately. With Push you still need a way of checking if the service is online/offline.
heliodor 4y ago
VictoriaMetrics is super simple to use and reliable. It's a breath of fresh air compared to running other tools.
oofnik 4y ago
We began a transition from a single behemoth Prometheus instance to Victoria Metrics about a month ago after evaluating a couple of similar tools. VM won out because of its relative operational simplicity and eye-popping benchmark results in resource usage comparisons. Documentation can be a little rough around the edges, but overall we're impressed so far.

marginalia_nu 4y ago

Batch jobs are arguably a lot better served with old-school logging than metrics. If you're using prometheus for alerts, I'll maybe go as far as considering incrementing a "the batch job failed"-counter a good idea. But all this is very much jamming a square peg into a round hole.

encoderer 4y ago
Also, there are tools like Cronitor that can help.

cduzz 4y ago

"Hey, I'm trying to figure out some business metrics related to our infrastructure and application performance / uptime; do we have the data anywhere?" "Sure, Bob, just write a promql query against this interface" https://news.ycombinator.com/item?id=1636348 https://news.ycombinator.com/item?id=1636348 (in case my point is too subtle: I believe that the value of long term metric retention is to support business decisions more than supporting availability related monitoring. The tools "business deciders" use to look at "the things they call metrics" are simply unable to look at data lodged in a prometheus silo. So the data you have for long term retention is simply unusable for the thing it's most valuable) (edited for clarity)

flerchin 4y ago
I use thanos for availability related concerns. eg, Some metric crossed the threshold for saturation, and I can go back as far as I need to see when the metric began saturating. We have seasonal business patterns that I can show the high water-mark for resource consumption. Thanos also allows a CQRS-type pattern if that's important to folks. Do I point the biz at it? Only when I'm trying to show them availability related metrics.
- deleted 4y ago
  [deleted]
gjulianm 4y ago
What tools do they use? It's not hard to build a dashboard in Grafana and show them there the metrics they want, in an easily-digestible format.
- cduzz 4y ago
  There's an entire universe of people making business decisions who live in excel and "BI" tools. If your thing doesn't speak ODBC / JDBC it's invisible. Further, the vast majority of workaday devs are stressed out and under tremendous pressure to thing all the things in their ticket queue. They're probably pretty fluent in their thinging language and likely also conversant in SQL. Asking them "hey learn this other query language, because" is just adding stress to them. Sure, some are going to rise above and promql things too, but for lots, it's telling them to "write a distributed map reduce query in erlang."

packetslave 4y ago

I would highly recommend looking at Cortex (or Grafana Labs' fork, Mimir) instead of Thanos. It doesn't rely on reading the Prometheus WAL (rather, it uses Prometheus's built-in remote-write functionality). This is both less fragile and has the benefit of being about to run Prometheus in stateless mode (or replace it entirely with something like Grafana Agent, vmagent, etc. Grafana Labs has done a proof of concept showing Mimir scaling to 1 billion active timeseries. There are also "real" users serving 700+ million metrics in production.

saifulwebid 4y ago
How do you “import” existing Prometheus metrics block into Cortex/Mimir using the remote-write, then?
- packetslave 4y ago
  For Thanos, you can (mostly) just copy the blocks from the existing s3 bucket into Mimir's bucket. There's some cleanup prep around external labels, but nothing too difficult, and the process is well-documented. I just finished migrating a year of Thanos metrics into Mimir without issue.