5 ms·

We've been using influx with much success. I just don't think Prometheus'es pull model is the right one for metrics especially in isolated sites like a DC. Has

We've been using influx with much success. I just don't think Prometheus'es pull model is the right one for metrics especially in isolated sites like a DC. Has anyone successfully migrated from influx to Prometheus? If so, why did you do that? What's better now?

IggleSniggle 3y ago
I thought Prometheus was a pull or push model? Although granted, I've spent very little time with it
- JimBlackwood 3y ago
  Prometheus will always pull metrics from a metrics exposing endpoint. However, Prometheus can then push metrics to anything that has the proper integrations for remote writing[0]. So you could run Prometheus in Agent Mode [1] in your DC to ingest metrics and push it to some central location. [0]: https://prometheus.io/docs/operating/integrations/#remote-en https://prometheus.io/docs/operating/integrations/#remote-en... [1]: https://prometheus.io/blog/2021/11/16/agent/ https://prometheus.io/blog/2021/11/16/agent/
  - gouthamve 3y ago
    Prometheus can also receive remote write requests, however, we recommend only writing metrics scraped by another Prometheus or the agent. The datamodel still has a few things that expect the metrics to have been scraped. If you try to use Prometheus as a native push system and push directly, it'll work but you might not have the best experience. https://prometheus.io/docs/prometheus/latest/querying/api/#remote-write-receiver https://prometheus.io/docs/prometheus/latest/querying/api/#r...
    - xgbi 3y ago
      Anyone with experience scaling Prometheus horizontally ? We are reaching the limits of our instance, memory and cpu wise, and I’m yet to choose between scaling it myself with sharding or using thanos/Victoria/cortex.
      - bboreham 3y ago
        If you want to query across the whole data set, use one of the other things. Prometheus has a "federation" option but there's not been any active work on it for years. It's basically the definition of Thanos - take a bunch of Prometheus and query across them. Plus long-term storage in S3. VictoriaMetrics, Cortex, Mimir are centralised data stores that accept data from multiple Prometheus, but you could also run headless agents scraping and sending the data. Note if you are on a version before 2.44, try upgrading. Prometheus slimmed down a bit. [I am a Prometheus and Mimir maintainer]
      - cassianoleal 3y ago
        I've beenthrough this song and dance. Did months-long PoCs (with live data, running next to the then-production Prometheus deployment) of Thanos, Cortex and Victoria Metrics. VM won hands down on pretty much all counts. It's easy and simple to operate and monitor, it scales really well and you can plan around how you want to partition and scale each component, it's incredibly cheap to run as performace is superior to the others, even when backed by spinning HDDs vs the other solutions on SSDs. It's especially easy to operate on Kubernetes using their CRDs and operators. I am not associated with Victoria Metrics in any way, just a happy user and sysadmin who ran it for a few years.
        mikepurvis 3y ago
        VictoriaMetrics was recommended to me by a contractor and I've been very happy with it as well. It does have an option to push in metrics, which I intend to use with transient environments like CI jobs and the like, though I haven't gotten there yet.
        cassianoleal 3y ago
        Yep, we used to use that in a few places. CI jobs, batch processes, etc. Prometheus has PushGateway which we also used before migrating to VM, but it had certain drawbacks (can't recall exactly what, sorry) that the new solution didn't.
      - FridgeSeal 3y ago
l33tman 3y ago
I've been using telegraf + influxdb + grafana for my last projects, never really had to tweak anything after uncommenting the right sections in the telegraf.conf.. is prometheus and associated tools kind of an alternative to that stack?
- valyala 3y ago
  Yes - take a look at VictoriaMetrics. https://docs.victoriametrics.com/guides/migrate-from-influx.html https://docs.victoriametrics.com/guides/migrate-from-influx....
valyala 3y ago
VictoriaMetrics has many users, who successfully migrated from InfluxDB. It supports data ingestion via Influx line protocol, so you can continue using Telegraf and sending the collected metrics to VictoriaMetrics instead of InfluxDB. You get the following benefits after the migration from InfluxDB to VictoriaMetrics: - Reduced memory usage by up to 10x [1]. - Reduced disk space usage. - Higher query performance. - Better query language than InfluxQL and Flux for typical queries over collected metrics [2]. - Compatibility with Prometheus ecosystem. See also InfluxDB -> VictoriaMetrics migration guide [3]. [1] https://valyala.medium.com/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893 https://valyala.medium.com/insert-benchmarks-with-inch-influ... [2] https://docs.victoriametrics.com/MetricsQL.html https://docs.victoriametrics.com/MetricsQL.html [3] https://docs.victoriametrics.com/guides/migrate-from-influx.html https://docs.victoriametrics.com/guides/migrate-from-influx....
- bitcharmer 3y ago
  Awesome, does victoria work with grafana?

IggleSniggle 3y ago

I thought Prometheus was a pull or push model? Although granted, I've spent very little time with it

JimBlackwood 3y ago
Prometheus will always pull metrics from a metrics exposing endpoint. However, Prometheus can then push metrics to anything that has the proper integrations for remote writing[0]. So you could run Prometheus in Agent Mode [1] in your DC to ingest metrics and push it to some central location. [0]: https://prometheus.io/docs/operating/integrations/#remote-en https://prometheus.io/docs/operating/integrations/#remote-en... [1]: https://prometheus.io/blog/2021/11/16/agent/ https://prometheus.io/blog/2021/11/16/agent/
- gouthamve 3y ago
  Prometheus can also receive remote write requests, however, we recommend only writing metrics scraped by another Prometheus or the agent. The datamodel still has a few things that expect the metrics to have been scraped. If you try to use Prometheus as a native push system and push directly, it'll work but you might not have the best experience. https://prometheus.io/docs/prometheus/latest/querying/api/#remote-write-receiver https://prometheus.io/docs/prometheus/latest/querying/api/#r...
  - xgbi 3y ago
    Anyone with experience scaling Prometheus horizontally ? We are reaching the limits of our instance, memory and cpu wise, and I’m yet to choose between scaling it myself with sharding or using thanos/Victoria/cortex.
    - bboreham 3y ago
      If you want to query across the whole data set, use one of the other things. Prometheus has a "federation" option but there's not been any active work on it for years. It's basically the definition of Thanos - take a bunch of Prometheus and query across them. Plus long-term storage in S3. VictoriaMetrics, Cortex, Mimir are centralised data stores that accept data from multiple Prometheus, but you could also run headless agents scraping and sending the data. Note if you are on a version before 2.44, try upgrading. Prometheus slimmed down a bit. [I am a Prometheus and Mimir maintainer]
    - cassianoleal 3y ago
      I've beenthrough this song and dance. Did months-long PoCs (with live data, running next to the then-production Prometheus deployment) of Thanos, Cortex and Victoria Metrics. VM won hands down on pretty much all counts. It's easy and simple to operate and monitor, it scales really well and you can plan around how you want to partition and scale each component, it's incredibly cheap to run as performace is superior to the others, even when backed by spinning HDDs vs the other solutions on SSDs. It's especially easy to operate on Kubernetes using their CRDs and operators. I am not associated with Victoria Metrics in any way, just a happy user and sysadmin who ran it for a few years.
      - mikepurvis 3y ago
        VictoriaMetrics was recommended to me by a contractor and I've been very happy with it as well. It does have an option to push in metrics, which I intend to use with transient environments like CI jobs and the like, though I haven't gotten there yet.
        cassianoleal 3y ago
        Yep, we used to use that in a few places. CI jobs, batch processes, etc. Prometheus has PushGateway which we also used before migrating to VM, but it had certain drawbacks (can't recall exactly what, sorry) that the new solution didn't.
    - FridgeSeal 3y ago

l33tman 3y ago

I've been using telegraf + influxdb + grafana for my last projects, never really had to tweak anything after uncommenting the right sections in the telegraf.conf.. is prometheus and associated tools kind of an alternative to that stack?

valyala 3y ago
Yes - take a look at VictoriaMetrics. https://docs.victoriametrics.com/guides/migrate-from-influx.html https://docs.victoriametrics.com/guides/migrate-from-influx....

valyala 3y ago

VictoriaMetrics has many users, who successfully migrated from InfluxDB. It supports data ingestion via Influx line protocol, so you can continue using Telegraf and sending the collected metrics to VictoriaMetrics instead of InfluxDB. You get the following benefits after the migration from InfluxDB to VictoriaMetrics: - Reduced memory usage by up to 10x [1]. - Reduced disk space usage. - Higher query performance. - Better query language than InfluxQL and Flux for typical queries over collected metrics [2]. - Compatibility with Prometheus ecosystem. See also InfluxDB -> VictoriaMetrics migration guide [3]. [1] https://valyala.medium.com/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893 https://valyala.medium.com/insert-benchmarks-with-inch-influ... [2] https://docs.victoriametrics.com/MetricsQL.html https://docs.victoriametrics.com/MetricsQL.html [3] https://docs.victoriametrics.com/guides/migrate-from-influx.html https://docs.victoriametrics.com/guides/migrate-from-influx....

bitcharmer 3y ago
Awesome, does victoria work with grafana?