3 ms·

So what does it signify that Prometheus is now under the CNCF? More development resources? I suppose it at least to some extent allays whatever fears there migh

by jabl 11y ago

So what does it signify that Prometheus is now under the CNCF? More development resources? I suppose it at least to some extent allays whatever fears there might have been that it will go "open core" in some ways, as e.g. InfluxDB recently.. That being said, what is the killer feature here? Looking at the docs, it seems nice, but collectd+influxdb+grafana seems to do the job for us at the moment..

sz4kerto 11y ago
InfluxDB _seems_ to be less stable, and also their monetization strategy might not appeal to some. I also appreciate the pull-style operation more and more (vs. the push-based operation of InfluxDB).
- gtirloni 11y ago
  Could you tell what you find better about the pull-style operation mode? My experience with pull-based monitoring tools comes from Zabbix/Nagios and it wasn't very pleasant (still had to configure clients extensively while keeping everything organized in a central database -- which is hard if your servers are coming and going all the time). We're currently using collectd+influxdb+grafana but it looks like Prometheus might be a better option in the future, since we plan to use Kubernetes more and more.
  - sagichmal 11y ago
    You're right in that pull only works well when your puller has a subscription to your service discovery system, so you don't need to manually configure targets when they change. Prometheus has SD plugins for major systems including Kubernetes.
infinotize 11y ago
Re: prometheus vs influx: There's a small thread with a few comments by the influxdb ceo and some Prometheus devs here [0], mainly on use case and philosophy. See also [1], although this is quite dated. [0] http://stackoverflow.com/questions/33350314/usecases-influxdb-vs-prometheus/ http://stackoverflow.com/questions/33350314/usecases-influxd... [1] https://prometheus.io/docs/introduction/comparison/#prometheus-vs.-influxdb https://prometheus.io/docs/introduction/comparison/#promethe...
bbrazil 11y ago
Prometheus developer here. > More development resources? Not directly. I expect the gains to be around the legal and organisational stuff you need as a project grows (e.g. trademark registration, formal governance), as well as publicity. > I suppose it at least to some extent allays whatever fears there might have been that it will go "open core" in some ways, as e.g. InfluxDB recently.. That was never really a risk, as Prometheus has never been a commercial effort controlled by any one company. > That being said, what is the killer feature here? Prometheus is perfect for monitoring any sort of operational timeseries, and is designed to do so in a way that scales both technically and organisationally. For example collectd is an on-host daemon that you have one of per machine. That may work when you're small, but with many services that will become a bottleneck as each need to be added and stragglers may become an issue (Twitter seem to have hit this, see "Lessons Learned" https://blog.twitter.com/2016/observability-at-twitter-technical-overview-part-ii https://blog.twitter.com/2016/observability-at-twitter-techn...). The Prometheus approach of one exporter per target scales far better. For another example if two different teams (say infra and database) have two different views of the world, with a collectd approach then the infra team likely wins. With Prometheus monitoring can be decentralised so infra can view a machine as mach1{rack="ch",datacenter="ma"} and database as mach1{owner="app2",env="canary"} each in their own Prometheus server. This gives teams more control by decentralising monitoring.
- jabl 11y ago
  Thanks for taking the time to answer. > I expect the gains to be around the legal and organisational stuff you need as a project grows (e.g. trademark registration, formal governance), as well as publicity. Good point. It's easy to forget the paperwork, but it certainly is important too! > Prometheus is perfect for monitoring any sort of operational timeseries, and is designed to do so in a way that scales both technically and organisationally. Umm, yeah, but well, isn't that the punchline of every monitoring project? ;-) > For example collectd is an on-host daemon that you have one of per machine. That may work when you're small, but with many services that will become a bottleneck as each need to be added and stragglers may become an issue (Twitter seem to have hit this, see "Lessons Learned" https://blog.twitter.com/2016/observability-at-twitter-techn... https://blog.twitter.com/2016/observability-at-twitter-techn...). The Prometheus approach of one exporter per target scales far better. I don't understand. Isn't that twitter blog essentially saying that they went from pull to push? Or is the main point here that they changed to have the agent on each host that collects everything on that host before transferring those metrics to the server (whether that's push or pull)? > For another example if two different teams (say infra and database) have two different views of the world, with a collectd approach then the infra team likely wins. With Prometheus monitoring can be decentralised so infra can view a machine as mach1{rack="ch",datacenter="ma"} and database as mach1{owner="app2",env="canary"} each in their own Prometheus server. This gives teams more control by decentralising monitoring. Good point. And I think this hits home in the sense that with our current monitoring setup the metrics we collect are really host-level, whereas for many metrics we'd rather have per-job (we're a HPC shop, so a bit different from the usual web 2.0-stuff) statistics. But I'm not sure Prometheus bends to this either, since IIUC each metric is stored in a separate file, and if we'd have a bunch of metrics for each job ID in the system, this wouldn't really work out, would it?

by jabl 11y ago

sz4kerto 11y ago

InfluxDB _seems_ to be less stable, and also their monetization strategy might not appeal to some. I also appreciate the pull-style operation more and more (vs. the push-based operation of InfluxDB).

gtirloni 11y ago
Could you tell what you find better about the pull-style operation mode? My experience with pull-based monitoring tools comes from Zabbix/Nagios and it wasn't very pleasant (still had to configure clients extensively while keeping everything organized in a central database -- which is hard if your servers are coming and going all the time). We're currently using collectd+influxdb+grafana but it looks like Prometheus might be a better option in the future, since we plan to use Kubernetes more and more.
- sagichmal 11y ago
  You're right in that pull only works well when your puller has a subscription to your service discovery system, so you don't need to manually configure targets when they change. Prometheus has SD plugins for major systems including Kubernetes.

infinotize 11y ago

Re: prometheus vs influx: There's a small thread with a few comments by the influxdb ceo and some Prometheus devs here [0], mainly on use case and philosophy. See also [1], although this is quite dated. [0] http://stackoverflow.com/questions/33350314/usecases-influxdb-vs-prometheus/ http://stackoverflow.com/questions/33350314/usecases-influxd... [1] https://prometheus.io/docs/introduction/comparison/#prometheus-vs.-influxdb https://prometheus.io/docs/introduction/comparison/#promethe...

bbrazil 11y ago

Prometheus developer here. > More development resources? Not directly. I expect the gains to be around the legal and organisational stuff you need as a project grows (e.g. trademark registration, formal governance), as well as publicity. > I suppose it at least to some extent allays whatever fears there might have been that it will go "open core" in some ways, as e.g. InfluxDB recently.. That was never really a risk, as Prometheus has never been a commercial effort controlled by any one company. > That being said, what is the killer feature here? Prometheus is perfect for monitoring any sort of operational timeseries, and is designed to do so in a way that scales both technically and organisationally. For example collectd is an on-host daemon that you have one of per machine. That may work when you're small, but with many services that will become a bottleneck as each need to be added and stragglers may become an issue (Twitter seem to have hit this, see "Lessons Learned" https://blog.twitter.com/2016/observability-at-twitter-technical-overview-part-ii https://blog.twitter.com/2016/observability-at-twitter-techn...). The Prometheus approach of one exporter per target scales far better. For another example if two different teams (say infra and database) have two different views of the world, with a collectd approach then the infra team likely wins. With Prometheus monitoring can be decentralised so infra can view a machine as mach1{rack="ch",datacenter="ma"} and database as mach1{owner="app2",env="canary"} each in their own Prometheus server. This gives teams more control by decentralising monitoring.

jabl 11y ago
Thanks for taking the time to answer. > I expect the gains to be around the legal and organisational stuff you need as a project grows (e.g. trademark registration, formal governance), as well as publicity. Good point. It's easy to forget the paperwork, but it certainly is important too! > Prometheus is perfect for monitoring any sort of operational timeseries, and is designed to do so in a way that scales both technically and organisationally. Umm, yeah, but well, isn't that the punchline of every monitoring project? ;-) > For example collectd is an on-host daemon that you have one of per machine. That may work when you're small, but with many services that will become a bottleneck as each need to be added and stragglers may become an issue (Twitter seem to have hit this, see "Lessons Learned" https://blog.twitter.com/2016/observability-at-twitter-techn... https://blog.twitter.com/2016/observability-at-twitter-techn...). The Prometheus approach of one exporter per target scales far better. I don't understand. Isn't that twitter blog essentially saying that they went from pull to push? Or is the main point here that they changed to have the agent on each host that collects everything on that host before transferring those metrics to the server (whether that's push or pull)? > For another example if two different teams (say infra and database) have two different views of the world, with a collectd approach then the infra team likely wins. With Prometheus monitoring can be decentralised so infra can view a machine as mach1{rack="ch",datacenter="ma"} and database as mach1{owner="app2",env="canary"} each in their own Prometheus server. This gives teams more control by decentralising monitoring. Good point. And I think this hits home in the sense that with our current monitoring setup the metrics we collect are really host-level, whereas for many metrics we'd rather have per-job (we're a HPC shop, so a bit different from the usual web 2.0-stuff) statistics. But I'm not sure Prometheus bends to this either, since IIUC each metric is stored in a separate file, and if we'd have a bunch of metrics for each job ID in the system, this wouldn't really work out, would it?