7 ms·

Show HN: Pup, real-time app metrics with Statsd

107 points by Vvick727 14y ago

silverlight 14y ago
Very cool! Is it possible to use an existing StatsD installation if we already have one running? Or is it pretty much an "all or nothing" package?
- olidb2 14y ago
  Pup will work with any existing statsd client - it just replaces the node.js statsd server. In other words, you can keep all your code and apps untouched (FYI I'm not the author, but a Datadog co-founder)
scottostler 14y ago
Nice! And that's an awesome logo.
- alq 14y ago
  Thanks!
- jongala 14y ago
  Double thanks :)
brokentone 14y ago
Excellent project! If I understand this right, it's an addition/replacement for StatsD that adds the graphing normally done in Graphite. I've tried to get the whole Graphite/StatsD stack setup before, and I could never get Graphite together quite right.
- josegonzalez 14y ago
  That's what it looks like. I'm not sure I really care much for it, other than the fact that it no longer appears to use the Graphite URL api. Personally, I'd rather have graphite-web running and then use an alternative frontend like Descartes[1] or Graphiti[2]. Still, it's pretty good that there is an alternative frontend, if nothing else but to expand the graphite ecosystem. [1] https://github.com/obfuscurity/descartes [2] https://github.com/paperlesspost/graphiti
- sciurus 14y ago
  Somewhat off-topic, but what value is there in using statsd rather than talking to carbon, the graphite backend, directly? Statsd receives metrics over udp; carbon-cache can receive them over tcp, udp, or amqp. Statsd aggregates metrics and flushed them a set intervals; if you want this, carbon-aggregator can do it. http://graphite.readthedocs.org/en/0.9.10/carbon-daemons.html http://graphite.readthedocs.org/en/0.9.10/carbon-daemons.htm...
  - achompas 14y ago
    Other, more experienced people can speak to this, but I'd argue that statsd has a useful ontology for classifying metrics based on what you want them to track. If you're watching a number go up, use a statsd.Counter. Want to track request arrival rates? Use statsd.Gauges. Want to record times for request fulfillment? Use statsd.Timers. Statsd isn't necessarily better or worse than carbon-cache via UDP, but it provides a handy solution for the above use cases.
axx 14y ago
Am i understanding this right: this is a kick in the butt for services like NewRelic? Does this always need to run on the same server as the Application? Or is it possible to run it on a different server and push the data to it? disclaimer: i have no experience with statsd/graphite etc.
- StavrosK 14y ago
  Statsd accepts data over UDP, so you can run it on whichever server you like, and the logging is asynchronous and takes a few microseconds (last time I counted). I haven't had a chance to use it in production extensively, but I tested it a bit and liked it very much.
- olidb2 14y ago
  So: - It doesn't need to run on the same server as the application, and you can have any number of apps reporting to it. The only limitation here is that apps communicate with Pup in UDP. - Our goal with pup was to make it super-easy to see app metrics. It is therefore much narrower in scope than services like New Relic or projects like Graphite. It's open-source, though, and you can take it anywhere you want. - We ourselves operate a service that can consume data from Pup and other sources, and provides metrics + events aggregation and correlation, fancy graphing, alerting, etc.. You can check-it out at http://datadoghq.com http://datadoghq.com
  - axx 14y ago
    So if i understand this right: - DogStatsD is running on my Server and is collecting data. - I can use "dogstatsd-ruby" to collect data from my (i.e.) Rails App. - DogStatsD reports Stats to your Service Datadog HQ (optional) - Pup is a small version of Datadog HQ that is running locally on my Server and connects directly to DogStatsD. - If i feel Pup doen't satisfy my needs, i can switch from Pup to Datadog HQ, get more functions and you my money? :) right?
    - olidb2 14y ago
      In a nutshell, yes. You can also contribute some of the features you need to pup if you are so inclined. We're on a mission to provide monitoring that doesn't suck, and we believe that making it easy and rewarding to instrument your app is an important step on the way.
amix 14y ago
This seems to be basically a collector that sends data to Datalog (a NON open-source service that costs $15/host/month). I recommend setting up Grahite+statsd (it's truly amazing and open-source realtime metrics setup).
- olidb2 14y ago
  This was developed for Datadog, yes. But Pup works fully stand-alone and is completely open-source - so you can take whatever you want / need from it. Pup has a narrower scope than Graphite (which we also use and love), but optimizes for shortest time to app metrics viz. (Oh and Datadog itself is all free up to 5 hosts - but remains completely optional)
  - achompas 14y ago
    > but optimizes for shortest time to app metrics viz. I'm not sure I understand what "shortest time to app metrics viz" means. Are you referring to the time Pup takes to create charts? Or the lowest time frequency (minutes, seconds etc.) it can handle? Graphite defaults to reporting at 1-minute intervals, and can report at a finer granularity (1 sec. intervals) if you tweak a few settings and set up a cluster. I'm just wondering how Pup differs in this respect.
    - olidb2 14y ago
      I meant it as a shortest time from "I wish I could see my app metrics" to actually seeing them. In other words, ease of set-up, lack of knobs, having metrics appear automatically. By default, pup aggregates statsd data in 10s intervals, and the UI refreshes it continuously.
      - achompas 14y ago
        Cool, sounds good!
- treeder 14y ago
  @amix, you clearly didn't read the post.
bgnm2000 14y ago
Too purpley.
blaines 14y ago
This is why the guys at DataDog are awesome. Thanks!
FreshCode 14y ago
Love the Pup branding! Awesome logo.
achompas 14y ago
Very cool app! I'm using Graphite a lot at work these days, so it's cool to see an alternative from a team based in NY.
otterley 14y ago
This looks potentially interesting, but I can't really evaluate it without thorough and accurate documentation. It would be nice if there were more than just instructions for downloading it and a trivial use case. Without it, your work's not done.
- katabatic 14y ago
  I was able to get in running against an existing statsd installation trivially, and explore it to my heart's content on my existing metrics. And I can look at the source - what more do you want? That was the easiest 3rd-party dashboard setup I've ever experienced.
  - olidb2 14y ago
    Thanks katabatic - reducing friction was exactly what we were going for.
  - otterley 14y ago
    If your needs are trivial, then it's probably adequate. Not everyone's needs are, though. I'm trying to evaluate it in the context of needing an enterprise-grade dashboard solution, with tens of thousands of metrics across thousands of hosts, and robust filtering/aggregation capability. To my knowledge nobody has really solved this problem yet in a satisfactory way, outside some closed-source solutions at big internet companies.
    - olidb2 14y ago
      otterley - for that, you can take a look at http://datadoghq.com http://datadoghq.com. We do all of the above. Our service isn't free beyond 5 hosts, but is quite a bit cheaper than rolling and running your own or flying blind and facing the consquences. It's a hard problem, and we're on a mission to solve it for companies that don't start with goo* or end in *ook.
      - otterley 14y ago
jondot 14y ago
Nice project, I love the UI. I solved the same problem in another way with: * collectd * collectd to graphite plugin (for server-level metrics, from all your servers) * graphite * statsd (for custom app metrics over UDP) * graphene - my own UI toolkit for graphite, also based on D3.js (http://github.com/jondot/graphene http://github.com/jondot/graphene) All in all, I got a better, more robust and detailed solution using these components. Using Chef, the time to set it up is very minimal.
- olidb2 14y ago
  Nice work with Graphene, indeed.
michaelmior 14y ago
I love the one line install. The solution certainly won't be perfect for everyone but having such a pain-free way to test it out is awesome.
- alq 14y ago
  Thanks! We really had the get-started-quickly scenario in mind.
  - michaelmior 14y ago
    One thing I find disappointing is that you can't delete severs once they've been added. Especially since they count towards your limit. Unless I'm missing something?
    - olidb2 14y ago
      They'll disappear automatically when you stop sending data about them (within a few hours) - we've done that for our customers who have elastic / rolling servers on AWS for ex. If that doesn't cut if for you, we're happy to help - let us know what you're trying to do!
      - michaelmior 14y ago
        That's kind of cool :) But what about the case where I have an important server that's down for a few hours. Do I lose all the data previously collected?
        olidb2 14y ago
        Nope - all the data stays there, and can be shown on dashboards. We just don't show the hosts that don't submit data in the host lists, and don't count them towards the total number of hosts charged for. Does that answer your question?
        14y ago
olidb2 14y ago
I should add that Team @datadoghq is on IRC in #datadog on irc.freenode.net Don't be shy if you have questions when running pup.
dbyrd 14y ago
Square has a good solution built on node & mongo called cube (http://square.github.com/cube/ http://square.github.com/cube/). To build custom UIs they wrote javascript libraries Crossfilter (http://square.github.com/crossfilter/ http://square.github.com/crossfilter/) & Cubism (http://square.github.com/cubism/ http://square.github.com/cubism/)
instakill 14y ago
What are some potential use cases for pup?
- alq 14y ago
  A no-hassle, real-time display of custom metrics for developers: 1 command to run and you can watch your app in action, as long as you instrument it with a statsd client (pretty much standard these days). So, no logs to parse to get any app metric, everything already graphed for you in real-time, quasi-nil setup. We chose statsd because it means that you don't have to change your code to get it to production. You can then rely on pup for display (but no historical data), or on your internal tool chain (e.g. graphite et al.) or give our service (http://www.datadoghq.com http://www.datadoghq.com) a try if you don't want to set your own stuff up.
hcarvalhoalves 14y ago
Does it only integrate with statsd, or could it aggregate metrics from collectd, for example? This could be an alternative to Graphite for graphing metrics because Graphite is a pain in the ass to setup (no proper installer), comes with a lot of bloat and is in a semi-broken state regarding features right now.
- olidb2 14y ago
  Pup will graph all the metrics Datadog does - it already collects OS metrics, and comes with a set of plug-ins for connecting to common apps, and it wouldn't be very difficult to connect collectd to it. Pup's feature-set is limited today, though, compared to graphite. You may also check-out the full http://datadoghq.com http://datadoghq.com service for more. It's SaaS, and free up to 5 servers.

107 points by Vvick727 14y ago

Very cool! Is it possible to use an existing StatsD installation if we already have one running? Or is it pretty much an "all or nothing" package?

olidb2 14y ago
Pup will work with any existing statsd client - it just replaces the node.js statsd server. In other words, you can keep all your code and apps untouched (FYI I'm not the author, but a Datadog co-founder)

scottostler 14y ago

Nice! And that's an awesome logo.

alq 14y ago
Thanks!
jongala 14y ago
Double thanks :)

brokentone 14y ago

Excellent project! If I understand this right, it's an addition/replacement for StatsD that adds the graphing normally done in Graphite. I've tried to get the whole Graphite/StatsD stack setup before, and I could never get Graphite together quite right.

josegonzalez 14y ago
That's what it looks like. I'm not sure I really care much for it, other than the fact that it no longer appears to use the Graphite URL api. Personally, I'd rather have graphite-web running and then use an alternative frontend like Descartes[1] or Graphiti[2]. Still, it's pretty good that there is an alternative frontend, if nothing else but to expand the graphite ecosystem. [1] https://github.com/obfuscurity/descartes [2] https://github.com/paperlesspost/graphiti
sciurus 14y ago
Somewhat off-topic, but what value is there in using statsd rather than talking to carbon, the graphite backend, directly? Statsd receives metrics over udp; carbon-cache can receive them over tcp, udp, or amqp. Statsd aggregates metrics and flushed them a set intervals; if you want this, carbon-aggregator can do it. http://graphite.readthedocs.org/en/0.9.10/carbon-daemons.html http://graphite.readthedocs.org/en/0.9.10/carbon-daemons.htm...
- achompas 14y ago
  Other, more experienced people can speak to this, but I'd argue that statsd has a useful ontology for classifying metrics based on what you want them to track. If you're watching a number go up, use a statsd.Counter. Want to track request arrival rates? Use statsd.Gauges. Want to record times for request fulfillment? Use statsd.Timers. Statsd isn't necessarily better or worse than carbon-cache via UDP, but it provides a handy solution for the above use cases.

axx 14y ago

Am i understanding this right: this is a kick in the butt for services like NewRelic? Does this always need to run on the same server as the Application? Or is it possible to run it on a different server and push the data to it? disclaimer: i have no experience with statsd/graphite etc.

StavrosK 14y ago
Statsd accepts data over UDP, so you can run it on whichever server you like, and the logging is asynchronous and takes a few microseconds (last time I counted). I haven't had a chance to use it in production extensively, but I tested it a bit and liked it very much.
olidb2 14y ago
So: - It doesn't need to run on the same server as the application, and you can have any number of apps reporting to it. The only limitation here is that apps communicate with Pup in UDP. - Our goal with pup was to make it super-easy to see app metrics. It is therefore much narrower in scope than services like New Relic or projects like Graphite. It's open-source, though, and you can take it anywhere you want. - We ourselves operate a service that can consume data from Pup and other sources, and provides metrics + events aggregation and correlation, fancy graphing, alerting, etc.. You can check-it out at http://datadoghq.com http://datadoghq.com
- axx 14y ago
  So if i understand this right: - DogStatsD is running on my Server and is collecting data. - I can use "dogstatsd-ruby" to collect data from my (i.e.) Rails App. - DogStatsD reports Stats to your Service Datadog HQ (optional) - Pup is a small version of Datadog HQ that is running locally on my Server and connects directly to DogStatsD. - If i feel Pup doen't satisfy my needs, i can switch from Pup to Datadog HQ, get more functions and you my money? :) right?
  - olidb2 14y ago
    In a nutshell, yes. You can also contribute some of the features you need to pup if you are so inclined. We're on a mission to provide monitoring that doesn't suck, and we believe that making it easy and rewarding to instrument your app is an important step on the way.

amix 14y ago

This seems to be basically a collector that sends data to Datalog (a NON open-source service that costs $15/host/month). I recommend setting up Grahite+statsd (it's truly amazing and open-source realtime metrics setup).

olidb2 14y ago
This was developed for Datadog, yes. But Pup works fully stand-alone and is completely open-source - so you can take whatever you want / need from it. Pup has a narrower scope than Graphite (which we also use and love), but optimizes for shortest time to app metrics viz. (Oh and Datadog itself is all free up to 5 hosts - but remains completely optional)
- achompas 14y ago
  > but optimizes for shortest time to app metrics viz. I'm not sure I understand what "shortest time to app metrics viz" means. Are you referring to the time Pup takes to create charts? Or the lowest time frequency (minutes, seconds etc.) it can handle? Graphite defaults to reporting at 1-minute intervals, and can report at a finer granularity (1 sec. intervals) if you tweak a few settings and set up a cluster. I'm just wondering how Pup differs in this respect.
  - olidb2 14y ago
    I meant it as a shortest time from "I wish I could see my app metrics" to actually seeing them. In other words, ease of set-up, lack of knobs, having metrics appear automatically. By default, pup aggregates statsd data in 10s intervals, and the UI refreshes it continuously.
    - achompas 14y ago
      Cool, sounds good!
treeder 14y ago
@amix, you clearly didn't read the post.

bgnm2000 14y ago

Too purpley.

blaines 14y ago

This is why the guys at DataDog are awesome. Thanks!

FreshCode 14y ago

Love the Pup branding! Awesome logo.

achompas 14y ago

Very cool app! I'm using Graphite a lot at work these days, so it's cool to see an alternative from a team based in NY.

otterley 14y ago

This looks potentially interesting, but I can't really evaluate it without thorough and accurate documentation. It would be nice if there were more than just instructions for downloading it and a trivial use case. Without it, your work's not done.

katabatic 14y ago
I was able to get in running against an existing statsd installation trivially, and explore it to my heart's content on my existing metrics. And I can look at the source - what more do you want? That was the easiest 3rd-party dashboard setup I've ever experienced.
- olidb2 14y ago
  Thanks katabatic - reducing friction was exactly what we were going for.
- otterley 14y ago
  If your needs are trivial, then it's probably adequate. Not everyone's needs are, though. I'm trying to evaluate it in the context of needing an enterprise-grade dashboard solution, with tens of thousands of metrics across thousands of hosts, and robust filtering/aggregation capability. To my knowledge nobody has really solved this problem yet in a satisfactory way, outside some closed-source solutions at big internet companies.
  - olidb2 14y ago
    otterley - for that, you can take a look at http://datadoghq.com http://datadoghq.com. We do all of the above. Our service isn't free beyond 5 hosts, but is quite a bit cheaper than rolling and running your own or flying blind and facing the consquences. It's a hard problem, and we're on a mission to solve it for companies that don't start with goo* or end in *ook.
    - otterley 14y ago

jondot 14y ago

Nice project, I love the UI. I solved the same problem in another way with: * collectd * collectd to graphite plugin (for server-level metrics, from all your servers) * graphite * statsd (for custom app metrics over UDP) * graphene - my own UI toolkit for graphite, also based on D3.js (http://github.com/jondot/graphene http://github.com/jondot/graphene) All in all, I got a better, more robust and detailed solution using these components. Using Chef, the time to set it up is very minimal.

olidb2 14y ago
Nice work with Graphene, indeed.

michaelmior 14y ago

I love the one line install. The solution certainly won't be perfect for everyone but having such a pain-free way to test it out is awesome.

alq 14y ago
Thanks! We really had the get-started-quickly scenario in mind.
- michaelmior 14y ago
  One thing I find disappointing is that you can't delete severs once they've been added. Especially since they count towards your limit. Unless I'm missing something?
  - olidb2 14y ago
    They'll disappear automatically when you stop sending data about them (within a few hours) - we've done that for our customers who have elastic / rolling servers on AWS for ex. If that doesn't cut if for you, we're happy to help - let us know what you're trying to do!
    - michaelmior 14y ago
      That's kind of cool :) But what about the case where I have an important server that's down for a few hours. Do I lose all the data previously collected?
      - olidb2 14y ago
        Nope - all the data stays there, and can be shown on dashboards. We just don't show the hosts that don't submit data in the host lists, and don't count them towards the total number of hosts charged for. Does that answer your question?
        14y ago

olidb2 14y ago

I should add that Team @datadoghq is on IRC in #datadog on irc.freenode.net Don't be shy if you have questions when running pup.

dbyrd 14y ago

Square has a good solution built on node & mongo called cube (http://square.github.com/cube/ http://square.github.com/cube/). To build custom UIs they wrote javascript libraries Crossfilter (http://square.github.com/crossfilter/ http://square.github.com/crossfilter/) & Cubism (http://square.github.com/cubism/ http://square.github.com/cubism/)

instakill 14y ago

What are some potential use cases for pup?

alq 14y ago
A no-hassle, real-time display of custom metrics for developers: 1 command to run and you can watch your app in action, as long as you instrument it with a statsd client (pretty much standard these days). So, no logs to parse to get any app metric, everything already graphed for you in real-time, quasi-nil setup. We chose statsd because it means that you don't have to change your code to get it to production. You can then rely on pup for display (but no historical data), or on your internal tool chain (e.g. graphite et al.) or give our service (http://www.datadoghq.com http://www.datadoghq.com) a try if you don't want to set your own stuff up.

hcarvalhoalves 14y ago

Does it only integrate with statsd, or could it aggregate metrics from collectd, for example? This could be an alternative to Graphite for graphing metrics because Graphite is a pain in the ass to setup (no proper installer), comes with a lot of bloat and is in a semi-broken state regarding features right now.

olidb2 14y ago
Pup will graph all the metrics Datadog does - it already collects OS metrics, and comes with a set of plug-ins for connecting to common apps, and it wouldn't be very difficult to connect collectd to it. Pup's feature-set is limited today, though, compared to graphite. You may also check-out the full http://datadoghq.com http://datadoghq.com service for more. It's SaaS, and free up to 5 servers.