11 ms·

Prometheus: An open-source service monitoring system and time series database

287 points by jjwiseman 12y ago

ggambetta 12y ago
"Those who cannot remember the Borgmon are doomed to repeat it" ;) Just kidding, this is looking really good, I hope to get some hands-on experience with it soon.
- deleted 12y ago
  [deleted]
  - danellis 12y ago
    What does that mean?
    - deleted 12y ago
      [deleted]
  - cespare 12y ago
    You mean Borgmon? The Google monitoring system that stopped being a secret years and years ago? (One example from 2 years ago: https://www.reddit.com/r/IAmA/comments/177267/we_are_the_google_site_reliability_team_we_make/c82y43e https://www.reddit.com/r/IAmA/comments/177267/we_are_the_goo...)
    - deleted 12y ago
      [deleted]
      - lifeisstillgood 12y ago
        What on earth? Google imposes non disclosure on ex-employees to mention by name in-house monitoring systems? (Presumably Borg-monitor?) That strikes me as a bit paranoid, not letting the name of a monitoring system be revealed. Am I missing something?
        deleted 12y ago
        [deleted]
        ithkuil 12y ago
        Once I heard a urban legend about a programmer being sued because he was using mickeyMouse as a variable name. I wouldn't be surprised if many of such cases of non-disclosure are to avoid hungry lawyers for trying to suck some blood out of rich companies. I guess the star trek franchise is owned by Paramount or something like this.
    - ggambetta 12y ago
      Thank you. I did, in fact, make sure the name and the fact that it is a monitoring system were already public (and from an official source) before mentioning them in this post.
- deleted 12y ago
  [deleted]
- chris_va 12y ago
  After seeing the Prometheus query language, I opened the comments looking for this quote :).
- thesnider 12y ago
  After working at a job that had a horrible patchwork of monitoring techniques (including at least two in-house systems), I was desperately pining for borgmon, actually. Never thought those words would come out of my mouth. This does seem to have addressed at least a couple of the issues with that system, in that its config language is sane, and its scrape format is well-defined and typed.
ddorian43 12y ago
Announced here: https://developers.soundcloud.com/blog/prometheus-monitoring-at-soundcloud https://developers.soundcloud.com/blog/prometheus-monitoring...
- bbrazil 12y ago
  The SoundCloud announcement is a great overview, there's also a series of blog posts I did on http://www.boxever.com/tag/monitoring http://www.boxever.com/tag/monitoring going into more depth with end-to-end examples.
kanwisher 12y ago
Would be interesting how this compares to InfluxDb
- sagichmal 12y ago
  http://prometheus.io/docs/introduction/comparison/#prometheus-vs.-influxdb http://prometheus.io/docs/introduction/comparison/#prometheu...
perlgeek 12y ago
This looks very interesting. From http://prometheus.io/docs/introduction/getting_started/ http://prometheus.io/docs/introduction/getting_started/ > Prometheus collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. I wonder if we'll see some plugins that allow data collection via snmp or nagios monitoring scripts or so. That would make it much easier to switch large existing monitoring systems over to prometheus.
- sagichmal 12y ago
  See Exporters for third-party systems[0] and Pushing metrics[1]. [0] http://prometheus.io/docs/instrumenting/exporters http://prometheus.io/docs/instrumenting/exporters [1] http://prometheus.io/docs/instrumenting/pushing http://prometheus.io/docs/instrumenting/pushing
- bbrazil 12y ago
  (One of the authors here) Just last night I wrote the https://github.com/prometheus/collectd_exporter https://github.com/prometheus/collectd_exporter, which you could do SNMP with. I do plan on writing a purpose-designed SNMP exporter in the next a few months to monitor my home network, if someone else doesn't get there first.
  - perlgeek 12y ago
    Awesome. I wondered if I should mention collectd as a possible source of data, and now you've already made it available.
- discordianfish 12y ago
  I've wrote a exporter for Docker, so if you have a Docker based infrastructure you get tons of metrics by using it: http://5pi.de/2015/01/26/monitor-docker-containers-with-prometheus/ http://5pi.de/2015/01/26/monitor-docker-containers-with-prom...
secure 12y ago
I used to use InfluxDB + a custom program to scrape HTTP endpoints and insert them into InfluxDB before. After playing around with Prometheus for a day or so, I’m convinced I need to switch to Prometheus :). The query language is so much better than what InfluxDB and others provide.
- jrv 12y ago
  (Prometheus author here) Thanks, that's awesome to hear! Feel free to also join us on #prometheus on freenode or our mailing list: https://groups.google.com/forum/#!forum/prometheus-developers https://groups.google.com/forum/#!forum/prometheus-developer...
reinhardt1053 12y ago
Promdash, the dashboard builder for Prometheus, is written in Ruby on Rails https://github.com/prometheus/promdash https://github.com/prometheus/promdash
e12e 12y ago
So... how does this compare to http://riemann.io/ http://riemann.io/ ? I just re-discovered riemann... and was thinking of pairing it with logstash and have a go. It would seem prometheus does something... similar?
- bbrazil 12y ago
  From my look at Riemann, it seems more aimed as an event monitoring system than a time-series monitoring system. You can (and many do) use Riemann as time-series monitoring system, my understanding is that Prometheus is a bit better for multi-dimensional labels. I could see Riemann being used as an alert manager on top of Prometheus, handling all the logic around de-duping of alerts and notification. Prometheus's own alert manager is considered experimental.
- brown9-2 12y ago
  Riemann has no concept of persistence, unless you forward events to some other system.
corford 12y ago
This looks great! Is an official Python client library on the roadmap?
- bbrazil 12y ago
  Yes, I've got something working at the moment. It needs cleanup, docs, unittests etc. If you want to help out, it's up at https://github.com/brian-brazil/client_python https://github.com/brian-brazil/client_python
clarkevans 12y ago
We've been looking for something like this, unfortunately the "pull" model won't work for us. We really need a push model so that our statistics server doesn't need access to every single producer. I see the pushgateway, but it seems deliberately not a centralized storage. I wonder what InfluxDB means by "distributed", that is, if I could use it to implement a push (where distributed agents push to a centralized metric server) model.
- bbrazil 12y ago
  (One of the Prometheus authors here) My understanding is that InfluxDB will work for your use case, as it's all push. > I see the pushgateway, but it seems deliberately not a centralized storage. Yeah, the primary use case for the pushgateway is service-level metrics for batch jobs. May I ask why accessing your producers is a problem? I know for Storm I was considering writing a plugin that'd hook into it's metrics system, that'd push to the pushgateway and Prometheus would then scrape. Not my preferred way of doing things, but some distributed processing systems need that approach when work assigment is dynamic and opaque across slaves.
  - clarkevans 12y ago
    Each node in our network has it's own access control permissions and organizational owner. It's easy enough to provide a centralized service with an end-point to push statistics from each node. In this case, organizational participation is "distributed". A pull model, while technically distributed, is organizationally centralized: I have to get each node's owner to grant me direct access. Politically, and for security reasons, that's not going to happen.
    - bbrazil 12y ago
      That'd be a challenge alright. One thing to consider is having each owner run their own Prometheus so that they can take advantage of all it's features, though it's a harder sell than "just push over here". http://www.boxever.com/push-vs-pull-for-monitoring http://www.boxever.com/push-vs-pull-for-monitoring looks at other bits of the push vs. pull question.
lxc 12y ago
What is a service monitoring system?
tinco 12y ago
In your architecture I see a single monolithic database server called 'Prometheus'. Does it shard? I can't find it in the documentation. You mention it's compatible with TSDB, why did you choose to implement your own backend, or is this a fork of TSDB? The tech does look awesome though!
- zacblazic 12y ago
  At the bottom of http://prometheus.io/docs/introduction/comparison/ http://prometheus.io/docs/introduction/comparison/ there is: "... Prometheus will be simpler to run initially, but will require explicit sharding once the capacity of a single node is exceeded."
- bbrazil 12y ago
  > Does it shard? Currently you can manually vertically shard, and in future we may have support for some horizontal sharding for when the targets of a given job are too many to be handled by a single server. You should only hit this when you get into thousands of targets. Our roadmap[1] includes hierarchical federation to support this use case. > You mention it's compatible with TSDB, why did you choose to implement your own backend, or is this a fork of TSDB? Prometheus isn't based on OpenTSDB, though it has the same data model. We've a comparison[2] in the docs. The core difference is that OpenTSDB is only a database, it doesn't offer a query language, graphing, client libraries and integration with other systems. We plan to offer OpenTSDB as a long-term storage backend for Prometheus. [1] http://prometheus.io/docs/introduction/roadmap/ http://prometheus.io/docs/introduction/roadmap/ [2] http://prometheus.io/docs/introduction/comparison/#prometheus-vs.-opentsdb http://prometheus.io/docs/introduction/comparison/#prometheu...
sre_ops 12y ago
Oh yes. Rather than fixing the real abet non very sexy problems ( i.e. how to scale the system and while still keeping it somewhat manageable ) we design yet another super cool system that has the same abet different in specificity short comings ( does not scale at a reasonable TCO/simplicity ) and get accolades for yet another front-end to a barely functioning back-end. Want to do something amazing? Get this into your requirement: Give a modern i7 with 64G RAM, 24x 250G 6Gbit/sec SSD and 2x 10G NICs achieve sustained ingest of 6Gbit/sec of metrics over 1 NIC and be able to accept commands spit out results of queries over the second NIC while being capable of running purge cleanups of the garbage. Oh and let clients use SQL to pull the data on the flow level Not only this would be impressive but it would actually optimize and simplify the monitoring in a great way.
mhax 12y ago
I'm a little wary of a monolithic solutions to monitoring/graphing/time series data storage - it gives me flashbacks of nagios/zabbix ;) I currently use a combination of sensu/graphite/grafana which allows a lot of flexability (albeit with some initial wrangling with the setup)
- gambiter 12y ago
  I'm not sure what's wrong with nagios or zabbix... I use them both in different capacities, and they are good at what they do. Of course a piecemeal solution is more flexible, but as you said, configuration can be a beast, so many people prefer monolithic systems.
xfalcox 12y ago
Guys, I've seen the libs for collecting services info, but how do I get OS level info, like load average, disk utilization, ram etc..? I suppose that there's a simple service that we need to deploy on each server? Any tips on this use case?
- bbrazil 12y ago
  We support that use case, here's a guide: http://www.boxever.com/monitoring-your-machines-with-prometheus http://www.boxever.com/monitoring-your-machines-with-prometh...
  - xfalcox 12y ago
    Thanks from a fellow brazilian!
  - wyldfire 12y ago
    That's great! > For machine monitoring Prometheus offers the Node exporter Is it possible for the frontend to utilize data from the cron-invoked sar/sadc that already covers much of this data? http://sebastien.godard.pagesperso-orange.fr/ http://sebastien.godard.pagesperso-orange.fr/
    - bbrazil 12y ago
      The default instrumentation that comes with node_exporter covers far more than what systat provides. Retrieving data from /proc at every scrape also gives you a more accurate timestamp, which helps reduce graph artifacts. As an aside, if you have machine-level cronjobs you want to expose metrics from you can use the textfile[1] module of the node_exporter, which reads in data from *.prom files in the same format as accepted by the Pushgateway. [1]https://github.com/prometheus/node_exporter/blob/master/collector/textfile.go https://github.com/prometheus/node_exporter/blob/master/coll... [2]https://github.com/prometheus/pushgateway https://github.com/prometheus/pushgateway
bhuga 12y ago
It's great to see new entrants into the monitoring and graphing space. These are problems that every company has, and yet there's no solution as widely accepted for monitoring, notifications or graphing as nginx is for a web server. Not that I'd do a better job, but every time I further configure our monitoring system, I get that feeling that we're missing something as an industry. It's a space with lots of tools that feel too big or too small; only graphite feels like it's doing one job fairly well. Alerting is the worst of it. Nagios and all the other alerting solutions I've played with feel just a bit off. They're either doing too much or carve out a role at boundaries that aren't quite right. This results in other systems wanting to do alerting, making it tough to compare tools. As an example, Prometheus has an alert manager under development: https://github.com/prometheus/alertmanager https://github.com/prometheus/alertmanager. Why isn't doing a great job at graphing enough of a goal? Is it a problem with the alerting tools, or is it a problem with boundaries between alerting, graphing, and notifications?
- bbrazil 12y ago
  > This results in other systems wanting to do alerting, making it tough to compare tools. I'd think you've hit the nail on the head, every monitoring system has to do a bit of everything such as alerting, machine monitoring, and graphing; rather than focusing on just doing one thing and doing it well. Prometheus comes with a powerful data model and query language, few existing systems support the same (e.g. Atlas and Riemann have the notion of a query language and labels) so we have to do a bit of everything to produce a coherent system. I think a separate common alert manager would be good if we could combine, currently in practice that role isn't being done as you rely on de-duplication in a tool such as Pagerduty with support for silencing alerts not being unified.
- jrv 12y ago
  Brian already hinted at this, but the main reason why we do this is to have coherency of the data model across the whole chain: client app -> Prometheus server -> alerting rules -> alert handling This allows e.g. initial dimensional labels from the clients to cleanly propagate all the way into alerts, and allows you to silence, aggregate, or route alerts by any label combination. If OpenTSDB-style dimensional metrics would have been more of a standard already, maybe that would be different.
mbell 12y ago
From the storage system docs: > which organizes sample data in chunks of constant size (1024 bytes payload). These chunks are then stored on disk in one file per time series. That is concerning, is this going to have the same problem with disk IO that graphite does? i.e. Every metric update requires a disk IO due to this one file per metric structure.
- falcolas 12y ago
  This was my first thought as well. Having one inode per metric measured can easily overwhelm some file systems, and the IO overhead on those disks gets silly (especially if it's not a SSD capable of constant-time writes against sparse data). Combine with the Cacti pull model, and I think a wait-and-see attitude is the best for this for now.
  - beorn7 12y ago
    We were tempted to implement our own data management in a couple of bigger files, but extensive testing of various modern file systems (XFS, ext4) resulted in the perhaps not surprising effect that those file systems are way better in managing the data than our home-grown solutions.
    - falcolas 12y ago
      So, the open source timeseries DBs (RRD Tool, Influx, and KairosDB), and others like sqlite or even InnoDB didn't make the cut? That surprises me. > file systems are way better in managing the data Except they're not managing data, they're just separating tables, to extend the DB metaphor. And you still run the chance of running out of inodes on a "modern" file system like ext4. After having briefly dug into the code, I'm particularly worried about the fact that instead of minimizing iops by only writing the relevant changes to the same file, Prometheus is constantly copying data from one file to the next, both to perform your checkpoints and to invalidate old data. That's a lot of iops for such basic (and frequently repeated) tasks. Still in wait-and-see mode.
      - 12y ago
SEJeff 12y ago
As a graphite maintainer, see my other post about problems with graphite: https://news.ycombinator.com/item?id=8908423 https://news.ycombinator.com/item?id=8908423 I'm super excited about prometheus, and can't wait to get some time to see if I can make it work on my rasberry pi. That being said, I'm also going to likely eventually work on a graphite-web / graphite-api pluggable backend to use prometheus as the backend storage platform. The more OSS metrics solutions, the better!
- bbrazil 12y ago
  Great! We're over in #prometheus on Freenode if there's anything we can help you with.
- falcolas 12y ago
  Before digging into converting the backend to Prometheus, you may want to have a look at their actual persistence method. What worries me is that they are still storing one metric per file, and are copying data from one file to the next to perform checkpoints or purges (which is admittedly an improvement over whisper's method of thrashing back and forth across a single file). They have the code to do in-place method of appending chunks, but in my admittedly brief foray through the code I could not actually find any use of it.
  - SEJeff 12y ago
    The graphite backend is pluggable. It isn't converting so much as adding support for. One of my favorite backends is Cyanite which ultimately stores the data in Cassandra.
  - jrv 12y ago
    Which storage/purge strategy would you recommend? (see also this thread for more discussion about the storage: https://news.ycombinator.com/item?id=8996832 https://news.ycombinator.com/item?id=8996832)
- mdeeks 12y ago
  A huge problem I'm having with graphite right now (which is making me look at influxdb, etc) is its inability to render graphs with lots and lots of lines. For example CPU usage across a cluster of of hundreds of machines almost always times out now. I essentially am graphing this: system.frontend-.cpu-0.cpu-used Where "frontend-" expands to 200 or so machines. I'm not entirely sure where the bottleneck is here. Would love to know if you have ideas. Is this a limitation of graphite-web itself? I have large graphite install of 20+ graphite carbon nodes running on SSDs and three additional graphite-web instances in front generating graphs. Ingesting something like 1 million metrics/min. Also I didn't realize there were still graphite maintainers (seriously. not trolling). There hasn't been a release of graphite in well over a year. I assumed it was dead by now. Any idea when we'll get a fresh release?
XorNot 12y ago
Huh, how fortuitous. I've been looking for this exact type of thing and HN gives me a great starting place to evaluate.
paulasmuth 12y ago
Shameless plug: This looks quite similar to FnordMetric, which also supports labels/multi dimensional time series, is StatsD wire compatible and supports SQL as a query language (so you won't have to learn yet another DSL)
simple10 12y ago
Looks really promising for smaller clusters. However, the pull/scraping model for stats could be problematic for larger scale. I've been experimenting with metrics collection using heka (node) -> amqp -> heka (aggregator) -> influxdb -> grafana. It works extremely well and scales nicely but requires writing lua code for anomaly detection and alerts – good or bad depending on your preference. I highly recommend considering Heka[1] for shipping logs to both ElasticSearch and InfluxDB if you need more scale and flexibility than Prometheus currently provides. [1] https://github.com/mozilla-services/heka https://github.com/mozilla-services/heka
- bbrazil 12y ago
  > However, the pull/scraping model for stats could be problematic for larger scale. From experience of similar systems at massive scale, I expect no scaling problems with pulling in and of itself. Indeed, there's some tactical operational options you get with pull that you don't have with push. See http://www.boxever.com/push-vs-pull-for-monitoring http://www.boxever.com/push-vs-pull-for-monitoring for my general thoughts on the issue. > InfluxDB InfluxDB seems best suited for event logging rather than systems monitoring. See also http://prometheus.io/docs/introduction/comparison/#prometheus-vs.-influxdb http://prometheus.io/docs/introduction/comparison/#prometheu...
  - simple10 12y ago
    Good point on push-vs-pull. I'm biased towards push because of microservices that behave like batch jobs. In effect, I'm using AMQP in a similar way as the Prometheus pushgateway. Agreed that InfluxDB is suited for event logging out of the box, but the March 2014 comparison of Influx is outdated IMO. I'm using Heka to send numeric time series data to Influx and full logs to ElasticSearch. It's possible to send full logs to non-clustered Influx in 0.8, but it's useful to split out concerns to different backends. I also like that Influx 0.9 dropped LevelDB support for BoltDB. There will be more opportunity for performance enhancements.
    - bbrazil 12y ago
      > the March 2014 comparison of Influx is outdated IMO. I think we expected that, feel free to add comments on the doc for things that are different now.
0xdeadbeefbabe 12y ago
While monitoring is obviously useful, I'm not understanding the obvious importance of a time series database. Can you collect enough measurements for the time series database to be useful? I worry that I would have lots of metrics to backup my wrong conclusions. I also worry that so much irrelevant data would drown out the relevant stuff, and cause the humans to ignore the system in time. I work with computers and servers, and not airplanes or trains.
- bbrazil 12y ago
  > Can you collect enough measurements for the time series database to be useful? Yes, instrument everything. See http://prometheus.io/docs/practices/instrumentation/#how-to-instrument http://prometheus.io/docs/practices/instrumentation/#how-to-... > I worry that I would have lots of metrics to backup my wrong conclusions. This is not so much a problem with time series as a question of epistemology. Well chosen consoles will help your initial analysis, and after that it's down to correct application of the scientific method. > I also worry that so much irrelevant data would drown out the relevant stuff I've seen many attempts by smart people to try and do automatic correlation of time series to aid debugging. It's never gotten out of the toy stage, as there is too much noise. You need to understand your metrics in order to use them.
rgj 12y ago
Is it me or is it impossible to navigate the documentation on an iPad?
- jrv 12y ago
  I don't own any Apple devices, so I can't test, but it works well on my Android phone (being Bootstrap and using responsive design). The top menu collapses into a button with three horizontal bars which shows the menu upon click. The documentation-specific navigation is always displayed expanded when in the docs section of the site, but the contents are displayed underneath it. What does it look like for you?
- discordianfish 12y ago
  On my iphone it looks good
zeus13i 12y ago
After reading this thread and comparing Influx and Prometheus, I've concluded that both look promising. I was going to go with Prometheus (as it's easier to get started with), but I was really put off by the 'promdash' dashboard - it uses iframes and depends on mysql. So I'm going with InfluxDB + Grafana and I'll keep an eye out for developments.
- bbrazil 12y ago
  You may want to look at the console templates, as they don't have any dependencies: http://prometheus.io/docs/visualization/consoles/ http://prometheus.io/docs/visualization/consoles/
- jrv 12y ago
  The only time where PromDash uses iframes is when you specifically add a "frame" (iframe) widget to your dashboard to embed arbitrary web content. Native Prometheus graphs and pie charts don't use iframes at all. Some kind of SQL backend is a dependency for now, however.
  - zeus13i 12y ago
    Ah! Good to know, thanks. Somehow I missed that.

287 points by jjwiseman 12y ago

"Those who cannot remember the Borgmon are doomed to repeat it" ;) Just kidding, this is looking really good, I hope to get some hands-on experience with it soon.

deleted 12y ago
[deleted]
- danellis 12y ago
  What does that mean?
  - deleted 12y ago
    [deleted]
- cespare 12y ago
  You mean Borgmon? The Google monitoring system that stopped being a secret years and years ago? (One example from 2 years ago: https://www.reddit.com/r/IAmA/comments/177267/we_are_the_google_site_reliability_team_we_make/c82y43e https://www.reddit.com/r/IAmA/comments/177267/we_are_the_goo...)
  - deleted 12y ago
    [deleted]
    - lifeisstillgood 12y ago
      What on earth? Google imposes non disclosure on ex-employees to mention by name in-house monitoring systems? (Presumably Borg-monitor?) That strikes me as a bit paranoid, not letting the name of a monitoring system be revealed. Am I missing something?
      - deleted 12y ago
        [deleted]
      - ithkuil 12y ago
        Once I heard a urban legend about a programmer being sued because he was using mickeyMouse as a variable name. I wouldn't be surprised if many of such cases of non-disclosure are to avoid hungry lawyers for trying to suck some blood out of rich companies. I guess the star trek franchise is owned by Paramount or something like this.
  - ggambetta 12y ago
    Thank you. I did, in fact, make sure the name and the fact that it is a monitoring system were already public (and from an official source) before mentioning them in this post.
deleted 12y ago
[deleted]
chris_va 12y ago
After seeing the Prometheus query language, I opened the comments looking for this quote :).
thesnider 12y ago
After working at a job that had a horrible patchwork of monitoring techniques (including at least two in-house systems), I was desperately pining for borgmon, actually. Never thought those words would come out of my mouth. This does seem to have addressed at least a couple of the issues with that system, in that its config language is sane, and its scrape format is well-defined and typed.

ddorian43 12y ago

Announced here: https://developers.soundcloud.com/blog/prometheus-monitoring-at-soundcloud https://developers.soundcloud.com/blog/prometheus-monitoring...

bbrazil 12y ago
The SoundCloud announcement is a great overview, there's also a series of blog posts I did on http://www.boxever.com/tag/monitoring http://www.boxever.com/tag/monitoring going into more depth with end-to-end examples.

kanwisher 12y ago

Would be interesting how this compares to InfluxDb

sagichmal 12y ago
http://prometheus.io/docs/introduction/comparison/#prometheus-vs.-influxdb http://prometheus.io/docs/introduction/comparison/#prometheu...

perlgeek 12y ago

This looks very interesting. From http://prometheus.io/docs/introduction/getting_started/ http://prometheus.io/docs/introduction/getting_started/ > Prometheus collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. I wonder if we'll see some plugins that allow data collection via snmp or nagios monitoring scripts or so. That would make it much easier to switch large existing monitoring systems over to prometheus.

sagichmal 12y ago
See Exporters for third-party systems[0] and Pushing metrics[1]. [0] http://prometheus.io/docs/instrumenting/exporters http://prometheus.io/docs/instrumenting/exporters [1] http://prometheus.io/docs/instrumenting/pushing http://prometheus.io/docs/instrumenting/pushing
bbrazil 12y ago
(One of the authors here) Just last night I wrote the https://github.com/prometheus/collectd_exporter https://github.com/prometheus/collectd_exporter, which you could do SNMP with. I do plan on writing a purpose-designed SNMP exporter in the next a few months to monitor my home network, if someone else doesn't get there first.
- perlgeek 12y ago
  Awesome. I wondered if I should mention collectd as a possible source of data, and now you've already made it available.
discordianfish 12y ago
I've wrote a exporter for Docker, so if you have a Docker based infrastructure you get tons of metrics by using it: http://5pi.de/2015/01/26/monitor-docker-containers-with-prometheus/ http://5pi.de/2015/01/26/monitor-docker-containers-with-prom...

secure 12y ago

I used to use InfluxDB + a custom program to scrape HTTP endpoints and insert them into InfluxDB before. After playing around with Prometheus for a day or so, I’m convinced I need to switch to Prometheus :). The query language is so much better than what InfluxDB and others provide.

jrv 12y ago
(Prometheus author here) Thanks, that's awesome to hear! Feel free to also join us on #prometheus on freenode or our mailing list: https://groups.google.com/forum/#!forum/prometheus-developers https://groups.google.com/forum/#!forum/prometheus-developer...

reinhardt1053 12y ago

Promdash, the dashboard builder for Prometheus, is written in Ruby on Rails https://github.com/prometheus/promdash https://github.com/prometheus/promdash

e12e 12y ago

So... how does this compare to http://riemann.io/ http://riemann.io/ ? I just re-discovered riemann... and was thinking of pairing it with logstash and have a go. It would seem prometheus does something... similar?

bbrazil 12y ago
From my look at Riemann, it seems more aimed as an event monitoring system than a time-series monitoring system. You can (and many do) use Riemann as time-series monitoring system, my understanding is that Prometheus is a bit better for multi-dimensional labels. I could see Riemann being used as an alert manager on top of Prometheus, handling all the logic around de-duping of alerts and notification. Prometheus's own alert manager is considered experimental.
brown9-2 12y ago
Riemann has no concept of persistence, unless you forward events to some other system.

corford 12y ago

This looks great! Is an official Python client library on the roadmap?

bbrazil 12y ago
Yes, I've got something working at the moment. It needs cleanup, docs, unittests etc. If you want to help out, it's up at https://github.com/brian-brazil/client_python https://github.com/brian-brazil/client_python

clarkevans 12y ago

We've been looking for something like this, unfortunately the "pull" model won't work for us. We really need a push model so that our statistics server doesn't need access to every single producer. I see the pushgateway, but it seems deliberately not a centralized storage. I wonder what InfluxDB means by "distributed", that is, if I could use it to implement a push (where distributed agents push to a centralized metric server) model.

bbrazil 12y ago
(One of the Prometheus authors here) My understanding is that InfluxDB will work for your use case, as it's all push. > I see the pushgateway, but it seems deliberately not a centralized storage. Yeah, the primary use case for the pushgateway is service-level metrics for batch jobs. May I ask why accessing your producers is a problem? I know for Storm I was considering writing a plugin that'd hook into it's metrics system, that'd push to the pushgateway and Prometheus would then scrape. Not my preferred way of doing things, but some distributed processing systems need that approach when work assigment is dynamic and opaque across slaves.
- clarkevans 12y ago
  Each node in our network has it's own access control permissions and organizational owner. It's easy enough to provide a centralized service with an end-point to push statistics from each node. In this case, organizational participation is "distributed". A pull model, while technically distributed, is organizationally centralized: I have to get each node's owner to grant me direct access. Politically, and for security reasons, that's not going to happen.
  - bbrazil 12y ago
    That'd be a challenge alright. One thing to consider is having each owner run their own Prometheus so that they can take advantage of all it's features, though it's a harder sell than "just push over here". http://www.boxever.com/push-vs-pull-for-monitoring http://www.boxever.com/push-vs-pull-for-monitoring looks at other bits of the push vs. pull question.

lxc 12y ago

What is a service monitoring system?

tinco 12y ago

In your architecture I see a single monolithic database server called 'Prometheus'. Does it shard? I can't find it in the documentation. You mention it's compatible with TSDB, why did you choose to implement your own backend, or is this a fork of TSDB? The tech does look awesome though!

zacblazic 12y ago
At the bottom of http://prometheus.io/docs/introduction/comparison/ http://prometheus.io/docs/introduction/comparison/ there is: "... Prometheus will be simpler to run initially, but will require explicit sharding once the capacity of a single node is exceeded."
bbrazil 12y ago
> Does it shard? Currently you can manually vertically shard, and in future we may have support for some horizontal sharding for when the targets of a given job are too many to be handled by a single server. You should only hit this when you get into thousands of targets. Our roadmap[1] includes hierarchical federation to support this use case. > You mention it's compatible with TSDB, why did you choose to implement your own backend, or is this a fork of TSDB? Prometheus isn't based on OpenTSDB, though it has the same data model. We've a comparison[2] in the docs. The core difference is that OpenTSDB is only a database, it doesn't offer a query language, graphing, client libraries and integration with other systems. We plan to offer OpenTSDB as a long-term storage backend for Prometheus. [1] http://prometheus.io/docs/introduction/roadmap/ http://prometheus.io/docs/introduction/roadmap/ [2] http://prometheus.io/docs/introduction/comparison/#prometheus-vs.-opentsdb http://prometheus.io/docs/introduction/comparison/#prometheu...

sre_ops 12y ago

Oh yes. Rather than fixing the real abet non very sexy problems ( i.e. how to scale the system and while still keeping it somewhat manageable ) we design yet another super cool system that has the same abet different in specificity short comings ( does not scale at a reasonable TCO/simplicity ) and get accolades for yet another front-end to a barely functioning back-end. Want to do something amazing? Get this into your requirement: Give a modern i7 with 64G RAM, 24x 250G 6Gbit/sec SSD and 2x 10G NICs achieve sustained ingest of 6Gbit/sec of metrics over 1 NIC and be able to accept commands spit out results of queries over the second NIC while being capable of running purge cleanups of the garbage. Oh and let clients use SQL to pull the data on the flow level Not only this would be impressive but it would actually optimize and simplify the monitoring in a great way.

mhax 12y ago

I'm a little wary of a monolithic solutions to monitoring/graphing/time series data storage - it gives me flashbacks of nagios/zabbix ;) I currently use a combination of sensu/graphite/grafana which allows a lot of flexability (albeit with some initial wrangling with the setup)

gambiter 12y ago
I'm not sure what's wrong with nagios or zabbix... I use them both in different capacities, and they are good at what they do. Of course a piecemeal solution is more flexible, but as you said, configuration can be a beast, so many people prefer monolithic systems.

xfalcox 12y ago

Guys, I've seen the libs for collecting services info, but how do I get OS level info, like load average, disk utilization, ram etc..? I suppose that there's a simple service that we need to deploy on each server? Any tips on this use case?

bbrazil 12y ago
We support that use case, here's a guide: http://www.boxever.com/monitoring-your-machines-with-prometheus http://www.boxever.com/monitoring-your-machines-with-prometh...
- xfalcox 12y ago
  Thanks from a fellow brazilian!
- wyldfire 12y ago
  That's great! > For machine monitoring Prometheus offers the Node exporter Is it possible for the frontend to utilize data from the cron-invoked sar/sadc that already covers much of this data? http://sebastien.godard.pagesperso-orange.fr/ http://sebastien.godard.pagesperso-orange.fr/
  - bbrazil 12y ago
    The default instrumentation that comes with node_exporter covers far more than what systat provides. Retrieving data from /proc at every scrape also gives you a more accurate timestamp, which helps reduce graph artifacts. As an aside, if you have machine-level cronjobs you want to expose metrics from you can use the textfile[1] module of the node_exporter, which reads in data from *.prom files in the same format as accepted by the Pushgateway. [1]https://github.com/prometheus/node_exporter/blob/master/collector/textfile.go https://github.com/prometheus/node_exporter/blob/master/coll... [2]https://github.com/prometheus/pushgateway https://github.com/prometheus/pushgateway

bhuga 12y ago

It's great to see new entrants into the monitoring and graphing space. These are problems that every company has, and yet there's no solution as widely accepted for monitoring, notifications or graphing as nginx is for a web server. Not that I'd do a better job, but every time I further configure our monitoring system, I get that feeling that we're missing something as an industry. It's a space with lots of tools that feel too big or too small; only graphite feels like it's doing one job fairly well. Alerting is the worst of it. Nagios and all the other alerting solutions I've played with feel just a bit off. They're either doing too much or carve out a role at boundaries that aren't quite right. This results in other systems wanting to do alerting, making it tough to compare tools. As an example, Prometheus has an alert manager under development: https://github.com/prometheus/alertmanager https://github.com/prometheus/alertmanager. Why isn't doing a great job at graphing enough of a goal? Is it a problem with the alerting tools, or is it a problem with boundaries between alerting, graphing, and notifications?

bbrazil 12y ago
> This results in other systems wanting to do alerting, making it tough to compare tools. I'd think you've hit the nail on the head, every monitoring system has to do a bit of everything such as alerting, machine monitoring, and graphing; rather than focusing on just doing one thing and doing it well. Prometheus comes with a powerful data model and query language, few existing systems support the same (e.g. Atlas and Riemann have the notion of a query language and labels) so we have to do a bit of everything to produce a coherent system. I think a separate common alert manager would be good if we could combine, currently in practice that role isn't being done as you rely on de-duplication in a tool such as Pagerduty with support for silencing alerts not being unified.
jrv 12y ago
Brian already hinted at this, but the main reason why we do this is to have coherency of the data model across the whole chain: client app -> Prometheus server -> alerting rules -> alert handling This allows e.g. initial dimensional labels from the clients to cleanly propagate all the way into alerts, and allows you to silence, aggregate, or route alerts by any label combination. If OpenTSDB-style dimensional metrics would have been more of a standard already, maybe that would be different.

mbell 12y ago

From the storage system docs: > which organizes sample data in chunks of constant size (1024 bytes payload). These chunks are then stored on disk in one file per time series. That is concerning, is this going to have the same problem with disk IO that graphite does? i.e. Every metric update requires a disk IO due to this one file per metric structure.

falcolas 12y ago
This was my first thought as well. Having one inode per metric measured can easily overwhelm some file systems, and the IO overhead on those disks gets silly (especially if it's not a SSD capable of constant-time writes against sparse data). Combine with the Cacti pull model, and I think a wait-and-see attitude is the best for this for now.
- beorn7 12y ago
  We were tempted to implement our own data management in a couple of bigger files, but extensive testing of various modern file systems (XFS, ext4) resulted in the perhaps not surprising effect that those file systems are way better in managing the data than our home-grown solutions.
  - falcolas 12y ago
    So, the open source timeseries DBs (RRD Tool, Influx, and KairosDB), and others like sqlite or even InnoDB didn't make the cut? That surprises me. > file systems are way better in managing the data Except they're not managing data, they're just separating tables, to extend the DB metaphor. And you still run the chance of running out of inodes on a "modern" file system like ext4. After having briefly dug into the code, I'm particularly worried about the fact that instead of minimizing iops by only writing the relevant changes to the same file, Prometheus is constantly copying data from one file to the next, both to perform your checkpoints and to invalidate old data. That's a lot of iops for such basic (and frequently repeated) tasks. Still in wait-and-see mode.
    - 12y ago

SEJeff 12y ago

As a graphite maintainer, see my other post about problems with graphite: https://news.ycombinator.com/item?id=8908423 https://news.ycombinator.com/item?id=8908423 I'm super excited about prometheus, and can't wait to get some time to see if I can make it work on my rasberry pi. That being said, I'm also going to likely eventually work on a graphite-web / graphite-api pluggable backend to use prometheus as the backend storage platform. The more OSS metrics solutions, the better!

bbrazil 12y ago
Great! We're over in #prometheus on Freenode if there's anything we can help you with.
falcolas 12y ago
Before digging into converting the backend to Prometheus, you may want to have a look at their actual persistence method. What worries me is that they are still storing one metric per file, and are copying data from one file to the next to perform checkpoints or purges (which is admittedly an improvement over whisper's method of thrashing back and forth across a single file). They have the code to do in-place method of appending chunks, but in my admittedly brief foray through the code I could not actually find any use of it.
- SEJeff 12y ago
  The graphite backend is pluggable. It isn't converting so much as adding support for. One of my favorite backends is Cyanite which ultimately stores the data in Cassandra.
- jrv 12y ago
  Which storage/purge strategy would you recommend? (see also this thread for more discussion about the storage: https://news.ycombinator.com/item?id=8996832 https://news.ycombinator.com/item?id=8996832)
mdeeks 12y ago
A huge problem I'm having with graphite right now (which is making me look at influxdb, etc) is its inability to render graphs with lots and lots of lines. For example CPU usage across a cluster of of hundreds of machines almost always times out now. I essentially am graphing this: system.frontend-.cpu-0.cpu-used Where "frontend-" expands to 200 or so machines. I'm not entirely sure where the bottleneck is here. Would love to know if you have ideas. Is this a limitation of graphite-web itself? I have large graphite install of 20+ graphite carbon nodes running on SSDs and three additional graphite-web instances in front generating graphs. Ingesting something like 1 million metrics/min. Also I didn't realize there were still graphite maintainers (seriously. not trolling). There hasn't been a release of graphite in well over a year. I assumed it was dead by now. Any idea when we'll get a fresh release?

XorNot 12y ago

Huh, how fortuitous. I've been looking for this exact type of thing and HN gives me a great starting place to evaluate.

paulasmuth 12y ago

Shameless plug: This looks quite similar to FnordMetric, which also supports labels/multi dimensional time series, is StatsD wire compatible and supports SQL as a query language (so you won't have to learn yet another DSL)

simple10 12y ago

Looks really promising for smaller clusters. However, the pull/scraping model for stats could be problematic for larger scale. I've been experimenting with metrics collection using heka (node) -> amqp -> heka (aggregator) -> influxdb -> grafana. It works extremely well and scales nicely but requires writing lua code for anomaly detection and alerts – good or bad depending on your preference. I highly recommend considering Heka[1] for shipping logs to both ElasticSearch and InfluxDB if you need more scale and flexibility than Prometheus currently provides. [1] https://github.com/mozilla-services/heka https://github.com/mozilla-services/heka

bbrazil 12y ago
> However, the pull/scraping model for stats could be problematic for larger scale. From experience of similar systems at massive scale, I expect no scaling problems with pulling in and of itself. Indeed, there's some tactical operational options you get with pull that you don't have with push. See http://www.boxever.com/push-vs-pull-for-monitoring http://www.boxever.com/push-vs-pull-for-monitoring for my general thoughts on the issue. > InfluxDB InfluxDB seems best suited for event logging rather than systems monitoring. See also http://prometheus.io/docs/introduction/comparison/#prometheus-vs.-influxdb http://prometheus.io/docs/introduction/comparison/#prometheu...
- simple10 12y ago
  Good point on push-vs-pull. I'm biased towards push because of microservices that behave like batch jobs. In effect, I'm using AMQP in a similar way as the Prometheus pushgateway. Agreed that InfluxDB is suited for event logging out of the box, but the March 2014 comparison of Influx is outdated IMO. I'm using Heka to send numeric time series data to Influx and full logs to ElasticSearch. It's possible to send full logs to non-clustered Influx in 0.8, but it's useful to split out concerns to different backends. I also like that Influx 0.9 dropped LevelDB support for BoltDB. There will be more opportunity for performance enhancements.
  - bbrazil 12y ago
    > the March 2014 comparison of Influx is outdated IMO. I think we expected that, feel free to add comments on the doc for things that are different now.

0xdeadbeefbabe 12y ago

While monitoring is obviously useful, I'm not understanding the obvious importance of a time series database. Can you collect enough measurements for the time series database to be useful? I worry that I would have lots of metrics to backup my wrong conclusions. I also worry that so much irrelevant data would drown out the relevant stuff, and cause the humans to ignore the system in time. I work with computers and servers, and not airplanes or trains.

bbrazil 12y ago
> Can you collect enough measurements for the time series database to be useful? Yes, instrument everything. See http://prometheus.io/docs/practices/instrumentation/#how-to-instrument http://prometheus.io/docs/practices/instrumentation/#how-to-... > I worry that I would have lots of metrics to backup my wrong conclusions. This is not so much a problem with time series as a question of epistemology. Well chosen consoles will help your initial analysis, and after that it's down to correct application of the scientific method. > I also worry that so much irrelevant data would drown out the relevant stuff I've seen many attempts by smart people to try and do automatic correlation of time series to aid debugging. It's never gotten out of the toy stage, as there is too much noise. You need to understand your metrics in order to use them.

rgj 12y ago

Is it me or is it impossible to navigate the documentation on an iPad?

jrv 12y ago
I don't own any Apple devices, so I can't test, but it works well on my Android phone (being Bootstrap and using responsive design). The top menu collapses into a button with three horizontal bars which shows the menu upon click. The documentation-specific navigation is always displayed expanded when in the docs section of the site, but the contents are displayed underneath it. What does it look like for you?
discordianfish 12y ago
On my iphone it looks good

zeus13i 12y ago

After reading this thread and comparing Influx and Prometheus, I've concluded that both look promising. I was going to go with Prometheus (as it's easier to get started with), but I was really put off by the 'promdash' dashboard - it uses iframes and depends on mysql. So I'm going with InfluxDB + Grafana and I'll keep an eye out for developments.

bbrazil 12y ago
You may want to look at the console templates, as they don't have any dependencies: http://prometheus.io/docs/visualization/consoles/ http://prometheus.io/docs/visualization/consoles/
jrv 12y ago
The only time where PromDash uses iframes is when you specifically add a "frame" (iframe) widget to your dashboard to embed arbitrary web content. Native Prometheus graphs and pie charts don't use iframes at all. Some kind of SQL backend is a dependency for now, however.
- zeus13i 12y ago
  Ah! Good to know, thanks. Somehow I missed that.