5 ms·

Graphiti: An Alternate Frontend for Graphite

61 points by vanessa 15y ago

latchkey 15y ago
Graphite is a great tool for quickly seeing what is going on. Generating the graphs was always a bit of a pain point for it. I must say, this is an amazing contribution. Thanks so much. (I'm the author of http://jmxtrans.googlecode.com http://jmxtrans.googlecode.com which allows you to very easily tie together Graphite & Java Management Extensions (JMX) for monitoring all of your JVM's.)
nwmcsween 15y ago
Why would I want to use graphite, a python based graphing solution to say rrdtool?
- latchkey 15y ago
  Not sure why you got voted down, so I'll vote you back up and answer your totally valid question. rrdtool is fine if you know your schema in advance, but that is somewhat of a hinderance because you don't always know it or want to have to define it. Graphite is great because it listens on a port an accepts data in a simple unstructured format. The format is: key : value (long) : timestamp Where key can be in dot notation so that you can have your data in a 'tree' like structure. This is a made up example: "ehcache.activemq.joins" or "ehcache.activemq.quits". The dashboard can then give you a nice tree to navigate through your data so that you can pin down specific areas that you'd like to graph. It also allows you to easily mix and match areas as overlays. Hope that explains things.
- sciurus 15y ago
  Well, you can use graphite as a frontend to your RRD files if you like. I'm using it to view data gathered by collectd and am planning to throw some RRD files generated by jmxtrans at it soon. If you were writing a tool that recorded time-series data, there's some explanation of why you might choose whisper over rrdtool at http://graphite.wikidot.com/whisper http://graphite.wikidot.com/whisper. I don't know of any other projects thats use whisper directly, though, and the creator of whisper is working on a replacement for it named ceres (http://graphite.wikidot.com/roadmap#toc0 http://graphite.wikidot.com/roadmap#toc0). The primary appeal is of the carbon daemon, which lets you send data to it over a super-simple protocol and takes care of persisting it for you.
  - latchkey 15y ago
    Given that jmxtrans has native support for writing directly to Graphite, why are you using it to write to rrd files? If you want those rrd files for other tools, the beauty of jmxtrans is that you can have multiple outputwriters so you can write to rrd AND graphite at the same time, with no loss in performance, since it happens in a multithreaded environment. Also note, writing to rrd files from jmxtrans is terribly inefficient. It spawns the rrdtool to do it because (unfortunately) the java implementation of rrd outputs files which are not compatible with rrdtool. (I'm the author of jmxtrans. Thanks for using my stuff!)
sciurus 15y ago
As this post recognizes, there are a lot of components bundled under the name Graphite: * There's whisper, the file format carbon uses to store time-series data. * There's carbon, the daemons that accept data over the network, combines them, and writes them to whisper files. * There's graphite, a django application that can read data from carbon files or RRD files. Graphite features several user interfaces of its own as well as an API to render the data as graphs or as numerical values. What I find most interesting is graphite's API's potential as an intermediary service between your metrics storage (whether its RRD, whisper, or some other format that you add support for) and your applications that need to consume those metrics (e.g. your monitoring system, your dashboards).
potch 15y ago
I've been working on a less-configurable sort of graphite dashboard for use as an ambient display- https://github.com/potch/statsdash https://github.com/potch/statsdash. Looks great on a tablet.
- mnutt 15y ago
  Cool, maybe consider adding a screenshot?
- sciurus 15y ago
  Another alternate graphite frontend worth following is Gdash. I like that its dashboard and graph configuration is done through YAML files. http://www.devco.net/archives/2011/10/08/gdash-graphite-dashboard.php http://www.devco.net/archives/2011/10/08/gdash-graphite-dash....
donspaulding 15y ago
Nice work! I've been noodling over an alternate backend metrics store using Redis, but probably closer to Mixpanel than Graphite in nature. I'd love to hear more about your pursuit of a redis solution, and why you abandoned it.
josephruscio 15y ago
Another alternative is to consider a service that handles storage/visualization/alerting/etc for your time-series data. I work for a startup that does exactly that and there are other options in this space as well. Would love any feedback you guys might have: https://metrics.librato.com https://metrics.librato.com
- amirmansour 15y ago
  Your service actually looks very interesting. Would you be able to tell us about your backend? How are you guys making it real-time? And how are you handeling storage?
  - josephruscio 15y ago
    Sure thing! Our backend is horizontally scalable and triply redundant by design. We have a web tier behind an ELB that runs our API (http://dev.librato.com/v1/post/metrics http://dev.librato.com/v1/post/metrics). It's implemented in Sinatra today. We'll probably switch to something more performant in the future, but for now it's cheaper to throw more instances behind the load-balancer and spend our time building out more features in the service. When metrics come into one of the API instances, they turn around and insert it into a Cassandra cluster that we're running spread across 3 different availability zones with an RF factor of 3, meaning your data is stored in 3 different availability zones. The "GET" calls to pull data out of our service go through the same API, so as soon as data is written to the cluster, it can be pulled back out. Hence the marketing term "realtime".
- latchkey 15y ago
  I'm so sorry, but I really don't see the value in spending $26.78/month for only 50 metrics. I could install Graphite on a small instance at AWS and get nearly unlimited metrics for a fraction of the cost. Never mind the fact that if I point your tool at a JVM, I can quickly get over 50 metrics (and double my cost to $53.57 since your slider only goes in increments of 50) just by looking at a single ehcache instance. Yes, your service is nice, your graphs are pretty, but at the end of the day, I think I'd have a hard time convincing my boss (me) that this is a valuable thing to spend a lot of money on. It seems like a lot of these 'monitoring' companies are springing up these days. I feel bad for you because I see that industry quickly being commoditized into who offers the lowest price. It is also not the easiest problem to solve because of the data storage and availability requirements. Anyway, I don't mean any ill will. I wish you the best in your business, I just don't see how it would work for me.
devinfoley 15y ago
I'm very excited by this. Graphite is an amazing tool, but the UI feels a bit outdated.

61 points by vanessa 15y ago

Graphite is a great tool for quickly seeing what is going on. Generating the graphs was always a bit of a pain point for it. I must say, this is an amazing contribution. Thanks so much. (I'm the author of http://jmxtrans.googlecode.com http://jmxtrans.googlecode.com which allows you to very easily tie together Graphite & Java Management Extensions (JMX) for monitoring all of your JVM's.)

nwmcsween 15y ago

Why would I want to use graphite, a python based graphing solution to say rrdtool?

latchkey 15y ago
Not sure why you got voted down, so I'll vote you back up and answer your totally valid question. rrdtool is fine if you know your schema in advance, but that is somewhat of a hinderance because you don't always know it or want to have to define it. Graphite is great because it listens on a port an accepts data in a simple unstructured format. The format is: key : value (long) : timestamp Where key can be in dot notation so that you can have your data in a 'tree' like structure. This is a made up example: "ehcache.activemq.joins" or "ehcache.activemq.quits". The dashboard can then give you a nice tree to navigate through your data so that you can pin down specific areas that you'd like to graph. It also allows you to easily mix and match areas as overlays. Hope that explains things.
sciurus 15y ago
Well, you can use graphite as a frontend to your RRD files if you like. I'm using it to view data gathered by collectd and am planning to throw some RRD files generated by jmxtrans at it soon. If you were writing a tool that recorded time-series data, there's some explanation of why you might choose whisper over rrdtool at http://graphite.wikidot.com/whisper http://graphite.wikidot.com/whisper. I don't know of any other projects thats use whisper directly, though, and the creator of whisper is working on a replacement for it named ceres (http://graphite.wikidot.com/roadmap#toc0 http://graphite.wikidot.com/roadmap#toc0). The primary appeal is of the carbon daemon, which lets you send data to it over a super-simple protocol and takes care of persisting it for you.
- latchkey 15y ago
  Given that jmxtrans has native support for writing directly to Graphite, why are you using it to write to rrd files? If you want those rrd files for other tools, the beauty of jmxtrans is that you can have multiple outputwriters so you can write to rrd AND graphite at the same time, with no loss in performance, since it happens in a multithreaded environment. Also note, writing to rrd files from jmxtrans is terribly inefficient. It spawns the rrdtool to do it because (unfortunately) the java implementation of rrd outputs files which are not compatible with rrdtool. (I'm the author of jmxtrans. Thanks for using my stuff!)

sciurus 15y ago

As this post recognizes, there are a lot of components bundled under the name Graphite: * There's whisper, the file format carbon uses to store time-series data. * There's carbon, the daemons that accept data over the network, combines them, and writes them to whisper files. * There's graphite, a django application that can read data from carbon files or RRD files. Graphite features several user interfaces of its own as well as an API to render the data as graphs or as numerical values. What I find most interesting is graphite's API's potential as an intermediary service between your metrics storage (whether its RRD, whisper, or some other format that you add support for) and your applications that need to consume those metrics (e.g. your monitoring system, your dashboards).

potch 15y ago

I've been working on a less-configurable sort of graphite dashboard for use as an ambient display- https://github.com/potch/statsdash https://github.com/potch/statsdash. Looks great on a tablet.

mnutt 15y ago
Cool, maybe consider adding a screenshot?
sciurus 15y ago
Another alternate graphite frontend worth following is Gdash. I like that its dashboard and graph configuration is done through YAML files. http://www.devco.net/archives/2011/10/08/gdash-graphite-dashboard.php http://www.devco.net/archives/2011/10/08/gdash-graphite-dash....

donspaulding 15y ago

Nice work! I've been noodling over an alternate backend metrics store using Redis, but probably closer to Mixpanel than Graphite in nature. I'd love to hear more about your pursuit of a redis solution, and why you abandoned it.

josephruscio 15y ago

Another alternative is to consider a service that handles storage/visualization/alerting/etc for your time-series data. I work for a startup that does exactly that and there are other options in this space as well. Would love any feedback you guys might have: https://metrics.librato.com https://metrics.librato.com

amirmansour 15y ago
Your service actually looks very interesting. Would you be able to tell us about your backend? How are you guys making it real-time? And how are you handeling storage?
- josephruscio 15y ago
  Sure thing! Our backend is horizontally scalable and triply redundant by design. We have a web tier behind an ELB that runs our API (http://dev.librato.com/v1/post/metrics http://dev.librato.com/v1/post/metrics). It's implemented in Sinatra today. We'll probably switch to something more performant in the future, but for now it's cheaper to throw more instances behind the load-balancer and spend our time building out more features in the service. When metrics come into one of the API instances, they turn around and insert it into a Cassandra cluster that we're running spread across 3 different availability zones with an RF factor of 3, meaning your data is stored in 3 different availability zones. The "GET" calls to pull data out of our service go through the same API, so as soon as data is written to the cluster, it can be pulled back out. Hence the marketing term "realtime".
latchkey 15y ago
I'm so sorry, but I really don't see the value in spending $26.78/month for only 50 metrics. I could install Graphite on a small instance at AWS and get nearly unlimited metrics for a fraction of the cost. Never mind the fact that if I point your tool at a JVM, I can quickly get over 50 metrics (and double my cost to $53.57 since your slider only goes in increments of 50) just by looking at a single ehcache instance. Yes, your service is nice, your graphs are pretty, but at the end of the day, I think I'd have a hard time convincing my boss (me) that this is a valuable thing to spend a lot of money on. It seems like a lot of these 'monitoring' companies are springing up these days. I feel bad for you because I see that industry quickly being commoditized into who offers the lowest price. It is also not the easiest problem to solve because of the data storage and availability requirements. Anyway, I don't mean any ill will. I wish you the best in your business, I just don't see how it would work for me.

devinfoley 15y ago

I'm very excited by this. Graphite is an amazing tool, but the UI feels a bit outdated.