7 ms·

Anyone use http://prometheus.io/ http://prometheus.io/ ? wdyt

by hendry 11y ago

MayMuncher 11y ago
I don't like it since the master basically does a curl on each server to get the info, so it doesn't work behind a firewall without tons of issues
- zimpenfish 11y ago
  That's the reason I haven't bothered with Prometheus - pull systems make no sense to me since you have to configure a single place with perfect* knowledge of your system rather than just pushing local knowledge to a collector. (Ok, you do need some knowledge at the parent if you want to raise alerts but you'd need that anyway.)
  - bbrazil 11y ago
    You need a place with all that information anyway, otherwise how do you alert on something being missing? If you can't easily do that with your existing infrastructure, you should fix that first. I've written about this at http://www.robustperception.io/you-look-good-have-you-lost-machines/ http://www.robustperception.io/you-look-good-have-you-lost-m...
    - zimpenfish 11y ago
      If a stream of data stops appearing, then you can alert. You don't need to pre-configure the existence of that stream (although if you might never get data from an object, this obviously is a failure you won't catch.) Your article is a good one but in my experience, many companies are still many years from being able to implement that kind of database:machine knowledge consistency.
      - bbrazil 11y ago
        You can only alert in that case if the data starts, and there's no transient issues preventing your monitoring working around the time it stops. Alerting based on state changes is fragile, it's better to compare against what you expect to see.
  - vruiz 11y ago
    You can leverage service discovery to make your configuration dynamic: http://prometheus.io/blog/2015/06/01/advanced-service-discovery/ http://prometheus.io/blog/2015/06/01/advanced-service-discov...
- robzienert 11y ago
  That's what the Prometheus push gateway is for. While prom itself is poll-based, you can extract data from servers behind a firewall by pushing metrics from an agent into the gateway. No crazy issues.
- bbrazil 11y ago
  The general advice is to run Prometheus behind the firewall, you want as few things between your monitoring and what you're monitoring as possible. This makes it more resilient.
- lobster_johnson 11y ago
  It supports push, though.
bovermyer 11y ago
We switched from using Riemann to Prometheus, and we're very happy with the decision. Prometheus has been much easier to work with.
uberneo 11y ago
Good blog about prometheus - https://developers.soundcloud.com/blog/prometheus-monitoring-at-soundcloud https://developers.soundcloud.com/blog/prometheus-monitoring...
lobster_johnson 11y ago
We're using Prometheus. Very happy, though the collection side of things aren't as smooth as we'd like it. The lack of a plugin system, and the reliance on HTTP to collect, means writing small collectors is a pain. You can't run 23 different daemons, each on their own port, to collect stats from things like PostgreSQL stats or RabbitMQ. We opted for the "text directory" way, where we populate a directory with .prom files that node_exporter automatically picks up. It's not ideal: It means a whole bunch of collectors run via Cron jobs, which themselves need to be monitored; it means if we remove a collector, we also have to clean up its .prom files; and in the end it meant we had to invent our own little plugin system in order to avoid writing a lot of boilerplate code needed to manage the different collectors we use. We'd love to share our collectors, but due to that last point, our collectors are less reusable than we'd like. Prometheus itself has been quite stable, but it still has some rough edges: * If anything goes wrong with its database files, it tends to just crash, and the only way out is to wipe the entire database (e.g., see [1]). * There's no way to do snapshots of the database. * The team is rather cavalier about backwards compatibility. We've experienced at least one version upgrade where they changed the database format and didn't provide any upgrade tools, so people were forced to start their metrics history from scratch. I know that it's pre-1.0, but still, they knew perfectly well that people were running it in production. The alert manager was also written from scratch recently, with a whole new config format. With several releases, every tool has had its command line flags changed ever so subtly, too. * The lack of packages (Debian/Ubuntu in our case) is also problematic. Fortunately I've got a script now that grabs a release and bundles a .deb from it, but I'd vastly preferred real releases. * No syslog support is not acceptable in this day and age. Our Upstart scripts spawns a "logger" subprocess to catch stderr. Not everyone is running under Docker. [1] https://github.com/prometheus/prometheus/issues/877 https://github.com/prometheus/prometheus/issues/877
- bbrazil 11y ago
  > The lack of a plugin system We have many ways to plugin to Prometheus across the ecosystem, the textfile collector you're using is one of them. > You can't run 23 different daemons, each on their own port, to collect stats from things like PostgreSQL stats or RabbitMQ. There's no fundamental challenge with this approach. If you've got good basic infrastructure, particularly configuration management, the rollout of each should be a small operational task. If it's a major challenge, then your problem probably isn't with the Prometheus architecture. > it means if we remove a collector, we also have to clean up its .prom files There's several problems arising from this approach, this is one of them. You can also expect odd artifacts in graphs. The textfile collector is only intended for machine-level metrics, by putting service level metrics in there you're missing out on a big win of Prometheus by thinking in terms of machines rather than services. Fighting against the architecture means you're not getting the maximum benefits from Prometheus, this would be easier with exporters and service discovery. > which themselves need to be monitored Are you aware that the node exporter exports the mtime of all the textfile collector files? That's there to make monitoring of them easier. > If anything goes wrong with its database files, it tends to just crash, and the only way out is to wipe the entire database As far as we're aware, the only way that happens is if you run out of disk space. If you've evidence otherwise please let us know, so we can prioritize accordingly. > We've experienced at least one version upgrade where they changed the database format and didn't provide any upgrade tools, so people were forced to start their metrics history from scratch. I know that it's pre-1.0, but still, they knew perfectly well that people were running it in production. We broke backwards compatibility in the storage format once, and there's no plans to do so again. The core developers who were all running it in production didn't see it as worthwhile to write a converter, and noone else stepped up. > The alert manager was also written from scratch recently, with a whole new config format. The old alertmanager has always been flagged as very experimental, as it was a functioning PoC. The rewrite was always been on the cards, and this came up regularly. This is all part of evolving the system to be better for everyone. If we tried to keep perfect backwards compatibility then we couldn't remove warts, bugs and misfeatures. We aren't afraid to deprecate where it makes sense to do so, and have transition plans where practical. > The lack of packages (Debian/Ubuntu in our case) is also problematic. There are packages in Debian proper, and nightlies at http://deb.robustperception.io/ http://deb.robustperception.io/ > No syslog support is not acceptable in this day and age. That's in the latest versions. The high level problem is that there's so many different ways to do logging that we can't sanely support them all. For every X there is someone who thinks it's essential.

Anyone use http://prometheus.io/ http://prometheus.io/ ? wdyt

by hendry 11y ago

Anyone use http://prometheus.io/ http://prometheus.io/ ? wdyt

I don't like it since the master basically does a curl on each server to get the info, so it doesn't work behind a firewall without tons of issues

zimpenfish 11y ago
That's the reason I haven't bothered with Prometheus - pull systems make no sense to me since you have to configure a single place with perfect* knowledge of your system rather than just pushing local knowledge to a collector. (Ok, you do need some knowledge at the parent if you want to raise alerts but you'd need that anyway.)
- bbrazil 11y ago
  You need a place with all that information anyway, otherwise how do you alert on something being missing? If you can't easily do that with your existing infrastructure, you should fix that first. I've written about this at http://www.robustperception.io/you-look-good-have-you-lost-machines/ http://www.robustperception.io/you-look-good-have-you-lost-m...
  - zimpenfish 11y ago
    If a stream of data stops appearing, then you can alert. You don't need to pre-configure the existence of that stream (although if you might never get data from an object, this obviously is a failure you won't catch.) Your article is a good one but in my experience, many companies are still many years from being able to implement that kind of database:machine knowledge consistency.
    - bbrazil 11y ago
      You can only alert in that case if the data starts, and there's no transient issues preventing your monitoring working around the time it stops. Alerting based on state changes is fragile, it's better to compare against what you expect to see.
- vruiz 11y ago
  You can leverage service discovery to make your configuration dynamic: http://prometheus.io/blog/2015/06/01/advanced-service-discovery/ http://prometheus.io/blog/2015/06/01/advanced-service-discov...
robzienert 11y ago
That's what the Prometheus push gateway is for. While prom itself is poll-based, you can extract data from servers behind a firewall by pushing metrics from an agent into the gateway. No crazy issues.
bbrazil 11y ago
The general advice is to run Prometheus behind the firewall, you want as few things between your monitoring and what you're monitoring as possible. This makes it more resilient.
lobster_johnson 11y ago
It supports push, though.

bovermyer 11y ago

We switched from using Riemann to Prometheus, and we're very happy with the decision. Prometheus has been much easier to work with.

uberneo 11y ago

Good blog about prometheus - https://developers.soundcloud.com/blog/prometheus-monitoring-at-soundcloud https://developers.soundcloud.com/blog/prometheus-monitoring...

lobster_johnson 11y ago

We're using Prometheus. Very happy, though the collection side of things aren't as smooth as we'd like it. The lack of a plugin system, and the reliance on HTTP to collect, means writing small collectors is a pain. You can't run 23 different daemons, each on their own port, to collect stats from things like PostgreSQL stats or RabbitMQ. We opted for the "text directory" way, where we populate a directory with .prom files that node_exporter automatically picks up. It's not ideal: It means a whole bunch of collectors run via Cron jobs, which themselves need to be monitored; it means if we remove a collector, we also have to clean up its .prom files; and in the end it meant we had to invent our own little plugin system in order to avoid writing a lot of boilerplate code needed to manage the different collectors we use. We'd love to share our collectors, but due to that last point, our collectors are less reusable than we'd like. Prometheus itself has been quite stable, but it still has some rough edges: * If anything goes wrong with its database files, it tends to just crash, and the only way out is to wipe the entire database (e.g., see [1]). * There's no way to do snapshots of the database. * The team is rather cavalier about backwards compatibility. We've experienced at least one version upgrade where they changed the database format and didn't provide any upgrade tools, so people were forced to start their metrics history from scratch. I know that it's pre-1.0, but still, they knew perfectly well that people were running it in production. The alert manager was also written from scratch recently, with a whole new config format. With several releases, every tool has had its command line flags changed ever so subtly, too. * The lack of packages (Debian/Ubuntu in our case) is also problematic. Fortunately I've got a script now that grabs a release and bundles a .deb from it, but I'd vastly preferred real releases. * No syslog support is not acceptable in this day and age. Our Upstart scripts spawns a "logger" subprocess to catch stderr. Not everyone is running under Docker. [1] https://github.com/prometheus/prometheus/issues/877 https://github.com/prometheus/prometheus/issues/877

bbrazil 11y ago
> The lack of a plugin system We have many ways to plugin to Prometheus across the ecosystem, the textfile collector you're using is one of them. > You can't run 23 different daemons, each on their own port, to collect stats from things like PostgreSQL stats or RabbitMQ. There's no fundamental challenge with this approach. If you've got good basic infrastructure, particularly configuration management, the rollout of each should be a small operational task. If it's a major challenge, then your problem probably isn't with the Prometheus architecture. > it means if we remove a collector, we also have to clean up its .prom files There's several problems arising from this approach, this is one of them. You can also expect odd artifacts in graphs. The textfile collector is only intended for machine-level metrics, by putting service level metrics in there you're missing out on a big win of Prometheus by thinking in terms of machines rather than services. Fighting against the architecture means you're not getting the maximum benefits from Prometheus, this would be easier with exporters and service discovery. > which themselves need to be monitored Are you aware that the node exporter exports the mtime of all the textfile collector files? That's there to make monitoring of them easier. > If anything goes wrong with its database files, it tends to just crash, and the only way out is to wipe the entire database As far as we're aware, the only way that happens is if you run out of disk space. If you've evidence otherwise please let us know, so we can prioritize accordingly. > We've experienced at least one version upgrade where they changed the database format and didn't provide any upgrade tools, so people were forced to start their metrics history from scratch. I know that it's pre-1.0, but still, they knew perfectly well that people were running it in production. We broke backwards compatibility in the storage format once, and there's no plans to do so again. The core developers who were all running it in production didn't see it as worthwhile to write a converter, and noone else stepped up. > The alert manager was also written from scratch recently, with a whole new config format. The old alertmanager has always been flagged as very experimental, as it was a functioning PoC. The rewrite was always been on the cards, and this came up regularly. This is all part of evolving the system to be better for everyone. If we tried to keep perfect backwards compatibility then we couldn't remove warts, bugs and misfeatures. We aren't afraid to deprecate where it makes sense to do so, and have transition plans where practical. > The lack of packages (Debian/Ubuntu in our case) is also problematic. There are packages in Debian proper, and nightlies at http://deb.robustperception.io/ http://deb.robustperception.io/ > No syslog support is not acceptable in this day and age. That's in the latest versions. The high level problem is that there's so many different ways to do logging that we can't sanely support them all. For every X there is someone who thinks it's essential.