7 ms·

I don't like it since the master basically does a curl on each server to get the info, so it doesn't work behind a firewall without tons of issues

zimpenfish 11y ago
That's the reason I haven't bothered with Prometheus - pull systems make no sense to me since you have to configure a single place with perfect* knowledge of your system rather than just pushing local knowledge to a collector. (Ok, you do need some knowledge at the parent if you want to raise alerts but you'd need that anyway.)
- bbrazil 11y ago
  You need a place with all that information anyway, otherwise how do you alert on something being missing? If you can't easily do that with your existing infrastructure, you should fix that first. I've written about this at http://www.robustperception.io/you-look-good-have-you-lost-machines/ http://www.robustperception.io/you-look-good-have-you-lost-m...
  - zimpenfish 11y ago
    If a stream of data stops appearing, then you can alert. You don't need to pre-configure the existence of that stream (although if you might never get data from an object, this obviously is a failure you won't catch.) Your article is a good one but in my experience, many companies are still many years from being able to implement that kind of database:machine knowledge consistency.
    - bbrazil 11y ago
      You can only alert in that case if the data starts, and there's no transient issues preventing your monitoring working around the time it stops. Alerting based on state changes is fragile, it's better to compare against what you expect to see.
- vruiz 11y ago
  You can leverage service discovery to make your configuration dynamic: http://prometheus.io/blog/2015/06/01/advanced-service-discovery/ http://prometheus.io/blog/2015/06/01/advanced-service-discov...
robzienert 11y ago
That's what the Prometheus push gateway is for. While prom itself is poll-based, you can extract data from servers behind a firewall by pushing metrics from an agent into the gateway. No crazy issues.
bbrazil 11y ago
The general advice is to run Prometheus behind the firewall, you want as few things between your monitoring and what you're monitoring as possible. This makes it more resilient.
lobster_johnson 11y ago
It supports push, though.

zimpenfish 11y ago

That's the reason I haven't bothered with Prometheus - pull systems make no sense to me since you have to configure a single place with perfect* knowledge of your system rather than just pushing local knowledge to a collector. (Ok, you do need some knowledge at the parent if you want to raise alerts but you'd need that anyway.)

bbrazil 11y ago
You need a place with all that information anyway, otherwise how do you alert on something being missing? If you can't easily do that with your existing infrastructure, you should fix that first. I've written about this at http://www.robustperception.io/you-look-good-have-you-lost-machines/ http://www.robustperception.io/you-look-good-have-you-lost-m...
- zimpenfish 11y ago
  If a stream of data stops appearing, then you can alert. You don't need to pre-configure the existence of that stream (although if you might never get data from an object, this obviously is a failure you won't catch.) Your article is a good one but in my experience, many companies are still many years from being able to implement that kind of database:machine knowledge consistency.
  - bbrazil 11y ago
    You can only alert in that case if the data starts, and there's no transient issues preventing your monitoring working around the time it stops. Alerting based on state changes is fragile, it's better to compare against what you expect to see.
vruiz 11y ago
You can leverage service discovery to make your configuration dynamic: http://prometheus.io/blog/2015/06/01/advanced-service-discovery/ http://prometheus.io/blog/2015/06/01/advanced-service-discov...

robzienert 11y ago

That's what the Prometheus push gateway is for. While prom itself is poll-based, you can extract data from servers behind a firewall by pushing metrics from an agent into the gateway. No crazy issues.

bbrazil 11y ago

The general advice is to run Prometheus behind the firewall, you want as few things between your monitoring and what you're monitoring as possible. This makes it more resilient.

lobster_johnson 11y ago

It supports push, though.