6 ms·

Ingest OpenTelemetry metrics with Prometheus natively

90 points by donutshop 3y ago

gouthamve 3y ago
Hi! Author of the PR here. As a project, Prometheus would like to become more OTel native and this is only the first of changes that are coming. I’d be happy to answer any questions you have.
- ranting-moth 3y ago
  Does it still apply that if my Prometheus goes down or network glitches then metrics for that period is lost forever?
  - gouthamve 3y ago
    Unfortunately, yes. The OTel Collector has plans to implement a WAL for the OTLP exporter and when it does, you should be resilient to upstream temporarily having issues.
    - SideburnsOfDoom 3y ago
      WAL = Write-Ahead Logging ?
      - gouthamve 3y ago
        Yes sorry
    - worserer 3y ago
      It does today. You have a retry queue and you can use a persistent storage for it.
    - ranting-moth 3y ago
      How will that affect timestamps, won't all the metrics have the timestamp as the time when Prometheus finally receives them?
  - kbouck 3y ago
    What are the merits of the prometheus approach versus one where events/metrics and their original timestamps can be preserved, stored (during temporary outages), then forwarded to backends when connectivity is reestablished?
- worserer 3y ago
  Do you have a roadmap of the changes you are looking to bring? Any changes to Otel you’re looking to add as well?
bitcharmer 3y ago
We've been using influx with much success. I just don't think Prometheus'es pull model is the right one for metrics especially in isolated sites like a DC. Has anyone successfully migrated from influx to Prometheus? If so, why did you do that? What's better now?
- IggleSniggle 3y ago
  I thought Prometheus was a pull or push model? Although granted, I've spent very little time with it
  - JimBlackwood 3y ago
    Prometheus will always pull metrics from a metrics exposing endpoint. However, Prometheus can then push metrics to anything that has the proper integrations for remote writing[0]. So you could run Prometheus in Agent Mode [1] in your DC to ingest metrics and push it to some central location. [0]: https://prometheus.io/docs/operating/integrations/#remote-en https://prometheus.io/docs/operating/integrations/#remote-en... [1]: https://prometheus.io/blog/2021/11/16/agent/ https://prometheus.io/blog/2021/11/16/agent/
    - gouthamve 3y ago
      Prometheus can also receive remote write requests, however, we recommend only writing metrics scraped by another Prometheus or the agent. The datamodel still has a few things that expect the metrics to have been scraped. If you try to use Prometheus as a native push system and push directly, it'll work but you might not have the best experience. https://prometheus.io/docs/prometheus/latest/querying/api/#remote-write-receiver https://prometheus.io/docs/prometheus/latest/querying/api/#r...
      - xgbi 3y ago
        Anyone with experience scaling Prometheus horizontally ? We are reaching the limits of our instance, memory and cpu wise, and I’m yet to choose between scaling it myself with sharding or using thanos/Victoria/cortex.
methou 3y ago
So no need for Jaegermeisters at work anymore!
wdb 3y ago
Nice, do I understand it correctly that this would mean there is a straight forward way to let Prometheus ingest the new histogram type without needing any new daemons (like otel-collector)? Currently, it doesn't appear the text format for pull metrics doesn't appear to support it.
- bboreham 3y ago
  The protobuf format supports native histograms. That’s the most straightforward way, if you have a client library for it. Go does, for instance. OpenTelemetry is push, so if you need pull and no new daemons this PR doesn’t help you.
  - wdb 3y ago
    Yeah, I have been reading up a bit. Only it's not clear to me if the protobuf format and text format can be mixed? Can one service A use text-format and service B use protobuf both scrapped by the same Prometheus sidecar?
    - bboreham 3y ago
      Yes. Scraping is done over http, and the format is negotiated via the ‘accept:’ header.
thefrozenone 3y ago
I've had an interesting time transitioning our project from OpenCensus to OpenTelemetry now that the former is EOL'd. We use the otel stackdriver output. Anyone have a refernce comparison between GCP cloud metrics vs. a prometheus monitoring stack?
- t-spawn 3y ago
  I did use stackdriver for quite a while before I moved to Mimir. TBH its great that you are still sticking to opentelemetry. Stackdriver as metric storage is not even a wise option in todays world give there are some really good TSDB providers SaaS or otherwise that would do a much better job. I moved away because of 2 primary reasons 1. The cost of stackdriver can add up with large-scale deployments or high-frequency metrics. It's essential to monitor and control usage to avoid unexpected billing. 2. I have experienced delays in metric updates, specifically at high frequency data. While the delays are usually minimal, they may not be ideal for some real-time monitoring use cases. FYI GCP on its own resources makes metrics available after 210s so you are always behind. Going the TSDB route to reliably run storage has worked for me. Also if this helps https://last9.io/blog/time-series-database-comparison/ https://last9.io/blog/time-series-database-comparison/
valyala 3y ago
Prometheus was known as a monitoring system, which promotes pull model over push model for metrics' collection. E.g. it is configured to discover scrape targets and then to scrape metrics from them at regular intervals - https://www.robustperception.io/its-easy-to-convert-pull-to-push/ https://www.robustperception.io/its-easy-to-convert-pull-to-... . It is interesting to see whether Prometheus will be transformed to multi-model (pull+push) monitoring system after the addition of Opentelemetry protocol. P.S. VictoriaMetrics, the Prometheus-like monitoring system I work on, also gained support for OpenTelemetry data ingestion in the release v1.92.0. https://docs.victoriametrics.com/CHANGELOG.html https://docs.victoriametrics.com/CHANGELOG.html

90 points by donutshop 3y ago

Hi! Author of the PR here. As a project, Prometheus would like to become more OTel native and this is only the first of changes that are coming. I’d be happy to answer any questions you have.

ranting-moth 3y ago
Does it still apply that if my Prometheus goes down or network glitches then metrics for that period is lost forever?
- gouthamve 3y ago
  Unfortunately, yes. The OTel Collector has plans to implement a WAL for the OTLP exporter and when it does, you should be resilient to upstream temporarily having issues.
  - SideburnsOfDoom 3y ago
    WAL = Write-Ahead Logging ?
    - gouthamve 3y ago
      Yes sorry
  - worserer 3y ago
    It does today. You have a retry queue and you can use a persistent storage for it.
  - ranting-moth 3y ago
    How will that affect timestamps, won't all the metrics have the timestamp as the time when Prometheus finally receives them?
- kbouck 3y ago
  What are the merits of the prometheus approach versus one where events/metrics and their original timestamps can be preserved, stored (during temporary outages), then forwarded to backends when connectivity is reestablished?
worserer 3y ago
Do you have a roadmap of the changes you are looking to bring? Any changes to Otel you’re looking to add as well?

bitcharmer 3y ago

We've been using influx with much success. I just don't think Prometheus'es pull model is the right one for metrics especially in isolated sites like a DC. Has anyone successfully migrated from influx to Prometheus? If so, why did you do that? What's better now?

IggleSniggle 3y ago
I thought Prometheus was a pull or push model? Although granted, I've spent very little time with it
- JimBlackwood 3y ago
  Prometheus will always pull metrics from a metrics exposing endpoint. However, Prometheus can then push metrics to anything that has the proper integrations for remote writing[0]. So you could run Prometheus in Agent Mode [1] in your DC to ingest metrics and push it to some central location. [0]: https://prometheus.io/docs/operating/integrations/#remote-en https://prometheus.io/docs/operating/integrations/#remote-en... [1]: https://prometheus.io/blog/2021/11/16/agent/ https://prometheus.io/blog/2021/11/16/agent/
  - gouthamve 3y ago
    Prometheus can also receive remote write requests, however, we recommend only writing metrics scraped by another Prometheus or the agent. The datamodel still has a few things that expect the metrics to have been scraped. If you try to use Prometheus as a native push system and push directly, it'll work but you might not have the best experience. https://prometheus.io/docs/prometheus/latest/querying/api/#remote-write-receiver https://prometheus.io/docs/prometheus/latest/querying/api/#r...
    - xgbi 3y ago
      Anyone with experience scaling Prometheus horizontally ? We are reaching the limits of our instance, memory and cpu wise, and I’m yet to choose between scaling it myself with sharding or using thanos/Victoria/cortex.

methou 3y ago

So no need for Jaegermeisters at work anymore!

wdb 3y ago

Nice, do I understand it correctly that this would mean there is a straight forward way to let Prometheus ingest the new histogram type without needing any new daemons (like otel-collector)? Currently, it doesn't appear the text format for pull metrics doesn't appear to support it.

bboreham 3y ago
The protobuf format supports native histograms. That’s the most straightforward way, if you have a client library for it. Go does, for instance. OpenTelemetry is push, so if you need pull and no new daemons this PR doesn’t help you.
- wdb 3y ago
  Yeah, I have been reading up a bit. Only it's not clear to me if the protobuf format and text format can be mixed? Can one service A use text-format and service B use protobuf both scrapped by the same Prometheus sidecar?
  - bboreham 3y ago
    Yes. Scraping is done over http, and the format is negotiated via the ‘accept:’ header.

thefrozenone 3y ago

I've had an interesting time transitioning our project from OpenCensus to OpenTelemetry now that the former is EOL'd. We use the otel stackdriver output. Anyone have a refernce comparison between GCP cloud metrics vs. a prometheus monitoring stack?

t-spawn 3y ago
I did use stackdriver for quite a while before I moved to Mimir. TBH its great that you are still sticking to opentelemetry. Stackdriver as metric storage is not even a wise option in todays world give there are some really good TSDB providers SaaS or otherwise that would do a much better job. I moved away because of 2 primary reasons 1. The cost of stackdriver can add up with large-scale deployments or high-frequency metrics. It's essential to monitor and control usage to avoid unexpected billing. 2. I have experienced delays in metric updates, specifically at high frequency data. While the delays are usually minimal, they may not be ideal for some real-time monitoring use cases. FYI GCP on its own resources makes metrics available after 210s so you are always behind. Going the TSDB route to reliably run storage has worked for me. Also if this helps https://last9.io/blog/time-series-database-comparison/ https://last9.io/blog/time-series-database-comparison/

valyala 3y ago

Prometheus was known as a monitoring system, which promotes pull model over push model for metrics' collection. E.g. it is configured to discover scrape targets and then to scrape metrics from them at regular intervals - https://www.robustperception.io/its-easy-to-convert-pull-to-push/ https://www.robustperception.io/its-easy-to-convert-pull-to-... . It is interesting to see whether Prometheus will be transformed to multi-model (pull+push) monitoring system after the addition of Opentelemetry protocol. P.S. VictoriaMetrics, the Prometheus-like monitoring system I work on, also gained support for OpenTelemetry data ingestion in the release v1.92.0. https://docs.victoriametrics.com/CHANGELOG.html https://docs.victoriametrics.com/CHANGELOG.html