4 ms·

Effortless monitoring with collectd, Graphite and Docker

72 points by kalessin 13y ago

gingerlime 13y ago
UDP is really great, but it also makes it extremely easy to spoof the source IP and flood your monitoring server with fake metrics. It's probably not a huge concern, but something to be aware of. See a blog post I wrote in the context of statsd/carbon[1] <shameless plug> Also, if you're looking for a nice open-source front-end to graphite, take a look at Giraffe[2]. It also has a collectd plugin[3] </shameless plug> [1]http://blog.gingerlime.com/2012/statsd-and-carbon-security/ http://blog.gingerlime.com/2012/statsd-and-carbon-security/ [2]http://giraffe.kenhub.com/ http://giraffe.kenhub.com/ [3]https://github.com/bflad/giraffe-collectd https://github.com/bflad/giraffe-collectd
- kalessin 13y ago
  Yeah, that's why collectd supports authentication and encryption (https://www.collectd.org/documentation/manpages/collectd.conf.5.shtml#plugin_network https://www.collectd.org/documentation/manpages/collectd.con...). I can't really speak about Graphtie since I'm less used to it; but in this case you can decide to not expose the Graphite port and only expose the collectd port.
  - gingerlime 13y ago
    Ah. I wasn't aware of this. Graphite/Carbon/Statsd are much more "basic" in that respect.
- sciurus 13y ago
  Good point. However, collectd's protocol is much more complex that carbon's; it can sign or encrypt the data to prevent spoofing. If spoofing is a concern, then instead of having collectd on your systems being monitored send directly to carbon, have them send to a collectd instance on your graphite server, and have that instance send to carbon on localhost. https://collectd.org/wiki/index.php/Networking_introduction https://collectd.org/wiki/index.php/Networking_introduction
hcarvalhoalves 13y ago
The biggest problem is setting up and upgrading Graphite, which is a pain and semi-broken right now. I've been successful using collectd + Librato, works great, specially since you can setup alerts effortlessly.
- gingerlime 13y ago
  Yes, setting up graphite can be confusing and painful. Looks like the author created a Docker app that should also make it easier. Another alternative and a shamless plug - I've created an open-source fabric script that automates the graphite install processes (on debian-based hosts). See https://github.com/gingerlime/graphite-fabric https://github.com/gingerlime/graphite-fabric
  - lsh123 13y ago
    Yep, I think it has too many dependencies and the architecture is over complicated. I've built a standalone stats collection/storage daemon that just works with any charts packaging - google charts, or whatever. Works great and doesn't require 10000 dependencies. https://github.com/lsh123/stats-rrdb https://github.com/lsh123/stats-rrdb
  - kawsper 13y ago
    I made a gist to show how to install it on the newest Ubuntu release: https://gist.github.com/kaspergrubbe/5792356 https://gist.github.com/kaspergrubbe/5792356 There is too many moving parts for my taste, but the web-interface is the worst.
  - hcarvalhoalves 13y ago
    I know, I've used your script in the past actually :)
- shykes 13y ago
  That's why they use docker: it automates the process of building and packaging Graphite, so you don't have to worry about it. $ docker run lopter/collectd-graphite You're done.
mdellabitta 13y ago
I don't understand this idea that you should build a metrics system from components rather than use one of the end-to-end projects out there like Zabbix. Can someone clue me in?
- otterley 13y ago
  I don't, either. And Graphite doesn't allow you to tag metrics for easy aggregation, which is a huge win if you're monitoring more than a few instances of a metric. (Nor does Zabbix, BTW.) I'm a reasonably happy Datadog customer.
  - jacobscott 13y ago
    Have you used any other cloud monitoring/metric/etc providers? If so, how do they compare to DataDog?
    - otterley 13y ago
      To my knowledge, none of them supports multi-dimensional metrics. So I had to cross them off my list.
  - TallboyOne 13y ago
    Datadog is very cool. What are some of the other solutions out there that are similar? I like how it mostly just integrates with anything I need, that makes things easy for a non-enterprise-level project.
- jlgreco 13y ago
  Barrier to entry. Zabbix is "enterprise" out its ass. Graphite, statsd, collectd are small simple systems that you can have mostly figured out in a few minutes. Meanwhile Zabbix's website makes it seem like the sole reason for that project to exist is to sell consulting and training gigs. Datadog seems like it might be slightly better, but I haven't spent much time looking at it.
  - 13y ago
sciurus 13y ago
To extend on this idea a little bit, once you have the data in graphite you can use it as input to an alerting system. As an example, here's a Nagios plugin I wrote to do that- https://github.com/sciurus/grallect https://github.com/sciurus/grallect
plasma 13y ago
I'd love to see a "fixed" version of statsd/graphite/etc that is painless to install, understand and use.

72 points by kalessin 13y ago

UDP is really great, but it also makes it extremely easy to spoof the source IP and flood your monitoring server with fake metrics. It's probably not a huge concern, but something to be aware of. See a blog post I wrote in the context of statsd/carbon[1] <shameless plug> Also, if you're looking for a nice open-source front-end to graphite, take a look at Giraffe[2]. It also has a collectd plugin[3] </shameless plug> [1]http://blog.gingerlime.com/2012/statsd-and-carbon-security/ http://blog.gingerlime.com/2012/statsd-and-carbon-security/ [2]http://giraffe.kenhub.com/ http://giraffe.kenhub.com/ [3]https://github.com/bflad/giraffe-collectd https://github.com/bflad/giraffe-collectd

kalessin 13y ago
Yeah, that's why collectd supports authentication and encryption (https://www.collectd.org/documentation/manpages/collectd.conf.5.shtml#plugin_network https://www.collectd.org/documentation/manpages/collectd.con...). I can't really speak about Graphtie since I'm less used to it; but in this case you can decide to not expose the Graphite port and only expose the collectd port.
- gingerlime 13y ago
  Ah. I wasn't aware of this. Graphite/Carbon/Statsd are much more "basic" in that respect.
sciurus 13y ago
Good point. However, collectd's protocol is much more complex that carbon's; it can sign or encrypt the data to prevent spoofing. If spoofing is a concern, then instead of having collectd on your systems being monitored send directly to carbon, have them send to a collectd instance on your graphite server, and have that instance send to carbon on localhost. https://collectd.org/wiki/index.php/Networking_introduction https://collectd.org/wiki/index.php/Networking_introduction

hcarvalhoalves 13y ago

The biggest problem is setting up and upgrading Graphite, which is a pain and semi-broken right now. I've been successful using collectd + Librato, works great, specially since you can setup alerts effortlessly.

gingerlime 13y ago
Yes, setting up graphite can be confusing and painful. Looks like the author created a Docker app that should also make it easier. Another alternative and a shamless plug - I've created an open-source fabric script that automates the graphite install processes (on debian-based hosts). See https://github.com/gingerlime/graphite-fabric https://github.com/gingerlime/graphite-fabric
- lsh123 13y ago
  Yep, I think it has too many dependencies and the architecture is over complicated. I've built a standalone stats collection/storage daemon that just works with any charts packaging - google charts, or whatever. Works great and doesn't require 10000 dependencies. https://github.com/lsh123/stats-rrdb https://github.com/lsh123/stats-rrdb
- kawsper 13y ago
  I made a gist to show how to install it on the newest Ubuntu release: https://gist.github.com/kaspergrubbe/5792356 https://gist.github.com/kaspergrubbe/5792356 There is too many moving parts for my taste, but the web-interface is the worst.
- hcarvalhoalves 13y ago
  I know, I've used your script in the past actually :)
shykes 13y ago
That's why they use docker: it automates the process of building and packaging Graphite, so you don't have to worry about it. $ docker run lopter/collectd-graphite You're done.

mdellabitta 13y ago

I don't understand this idea that you should build a metrics system from components rather than use one of the end-to-end projects out there like Zabbix. Can someone clue me in?

otterley 13y ago
I don't, either. And Graphite doesn't allow you to tag metrics for easy aggregation, which is a huge win if you're monitoring more than a few instances of a metric. (Nor does Zabbix, BTW.) I'm a reasonably happy Datadog customer.
- jacobscott 13y ago
  Have you used any other cloud monitoring/metric/etc providers? If so, how do they compare to DataDog?
  - otterley 13y ago
    To my knowledge, none of them supports multi-dimensional metrics. So I had to cross them off my list.
- TallboyOne 13y ago
  Datadog is very cool. What are some of the other solutions out there that are similar? I like how it mostly just integrates with anything I need, that makes things easy for a non-enterprise-level project.
jlgreco 13y ago
Barrier to entry. Zabbix is "enterprise" out its ass. Graphite, statsd, collectd are small simple systems that you can have mostly figured out in a few minutes. Meanwhile Zabbix's website makes it seem like the sole reason for that project to exist is to sell consulting and training gigs. Datadog seems like it might be slightly better, but I haven't spent much time looking at it.
- 13y ago

sciurus 13y ago

To extend on this idea a little bit, once you have the data in graphite you can use it as input to an alerting system. As an example, here's a Nagios plugin I wrote to do that- https://github.com/sciurus/grallect https://github.com/sciurus/grallect

plasma 13y ago

I'd love to see a "fixed" version of statsd/graphite/etc that is painless to install, understand and use.