6 ms·

Batsd: 37 Signals' Ruby Statsd implementation persisting to Redis

69 points by uggedal 14y ago

taylorbuley 14y ago
What does "'wireline' compatibility" mean?
- noahnoahnoah 14y ago
  We don't use the same backend as Etsy's original implementation, but you should be able to take any script, etc. that emits statsd measurements for use with the original implementation and point it at Batsd. This means that the 50+ statsd clients that are out there, as well as lots of custom instrumentation, should "just work", even though it's stored and accessed in different ways after it's received.
scoot 14y ago
So what can I use this for? It's not obvious from the docs on github, and I had a quick look at the doc for statsd as well and it doesn't shed much light either. (and shouldn't it be ratsd? :)
- mattyb 14y ago
  The Etsy blog post that introduced StatsD is helpful: http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/ http://codeascraft.etsy.com/2011/02/15/measure-anything-meas...
  - deleted 14y ago
    [deleted]
- noahnoahnoah 14y ago
  (I work for 37signals and wrote batsd) This is really just one piece in a bigger set of things to track performance, usage, etc. You can think of it as: Emitters --> Statsd (or in this case, Batsd) --> dashboards, alerts, etc. We have emitters coming from Nginx, HAproxy, bluepill, postfix, etc. log files, a gem within all of our Rails apps, and a variety of other scripts that gather data. Those all point to batsd, which aggregates and stores them. We then extract the data into graphs on our dashboard, and use it extensively for Nagios alerting as well. There's a basic sample client included in this repository that we use for those purposes, though you're right, it just gets you raw numbers out of the box. We're planning on releasing more of both the "emitters" that gather data, as well as a major part of our graphing/dashboard interface "soon". And point well taken about making it more obvious how to get started and what you can use it for. I'll work on improving the documentation.
  - scoot 14y ago
    Thank you, that's very helpful. Looking forward to the emitters and dashboard whenever they're ready - I suspect they will help drive adoption of Batsd and encourage development of additional emitters.
    - noahnoahnoah 14y ago
      Definitely. There's a teaser screenshot of Flyash (the big, reusable chunk of our dashboard) towards the bottom of http://37signals.com/svn/posts/3091-pssst-your-rails-application-has-a-secret-to-tell-you http://37signals.com/svn/posts/3091-pssst-your-rails-applica... (that post also details some of the major emitter components we use).
      - scoot 14y ago
        "457,739 different metrics in Flyash" Oh...kay... Flyash looks very nice from the screenshots, but I'm interested to learn how you solve the discoverability problem with that much data. (Too much of a good thing?)
      - mtalantikite 14y ago
        Looks great, can't wait to see it open sourced. I've been dealing with the clunkiness of statsd/data -> graphite -> graphene for a dashboard, and have more than a handful of times almost started writing exactly what it looks like you already have done. Any idea when we'll be able to use/contribute to Flyash?
        noahnoahnoah 14y ago
        A couple of weeks, probably, depending on how much I feel like working on it. It was designed to be modular and easily extracted, but still needs some cleanup work and has a few nasty bugs I'd like to fix first.
  - julian37 14y ago
    Could you explain briefly why you chose to write a replacement for statsd, rather than improve on it? What aspects of statsd were you not happy with? (I don't have a horse in this race, I haven't used statsd before -- but I am planning to deploy some sort of statistics gathering soon and I wonder why I would choose your implementation over Etsy's, apart from the obvious appeal of the 37signals brand.)
caiusdurling 14y ago
YEAAAAAAAAH!
uggedal 14y ago
Those interested in lower memory consumption and efficiency should take a look at https://github.com/armon/statsite https://github.com/armon/statsite another statsd "clone". Originally a Python implementation statsite is now rewritten in C.
- taligent 14y ago
  I wouldn't exactly call that a clone since it doesn't support key features of statsd (e.g. gauges) and has unique features of its own (e.g. key/value pairs). Looks brilliant otherwise.
  - DavidAbrams 14y ago
    +1 for saying "key/value" instead of "key-value".
  - armon 14y ago
    It is actually just a difference of naming. Gauges in statsd operate the same as kv data in statsite. Originally, both projects referred to them as kv, this seems to be a more recent change in statsd.
bri3d 14y ago
Similar, but backed by Graphite: https://github.com/mojodna/metricsd https://github.com/mojodna/metricsd It's awesome to see the community standardizing around a simple way of transporting measurements into a storage and analytics system.
richyzhang 14y ago
I would like to see Flyash you mentioned in the readme as soon as possible. For a monitor system, it is a crucial part.
nwmcsween 14y ago
Why should I use this over http://collectd.org http://collectd.org?
- mattyb 14y ago
  You don't use this over collectd; you can use BatsD with it. These tools are complementary. collectd is very good at collecting system-level metrics and sending them somewhere. Batsd is a system for receiving those metrics and storing them. (Note that collectd has a graphite plugin, but not one for (B|St)atsD. You'd need to write a proxy.)
igs123 14y ago
Content was really very informative. From http://www.indiagiftservices.com http://www.indiagiftservices.com

69 points by uggedal 14y ago

taylorbuley 14y ago

What does "'wireline' compatibility" mean?

noahnoahnoah 14y ago
We don't use the same backend as Etsy's original implementation, but you should be able to take any script, etc. that emits statsd measurements for use with the original implementation and point it at Batsd. This means that the 50+ statsd clients that are out there, as well as lots of custom instrumentation, should "just work", even though it's stored and accessed in different ways after it's received.

scoot 14y ago

So what can I use this for? It's not obvious from the docs on github, and I had a quick look at the doc for statsd as well and it doesn't shed much light either. (and shouldn't it be ratsd? :)

mattyb 14y ago
The Etsy blog post that introduced StatsD is helpful: http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/ http://codeascraft.etsy.com/2011/02/15/measure-anything-meas...
- deleted 14y ago
  [deleted]
noahnoahnoah 14y ago
(I work for 37signals and wrote batsd) This is really just one piece in a bigger set of things to track performance, usage, etc. You can think of it as: Emitters --> Statsd (or in this case, Batsd) --> dashboards, alerts, etc. We have emitters coming from Nginx, HAproxy, bluepill, postfix, etc. log files, a gem within all of our Rails apps, and a variety of other scripts that gather data. Those all point to batsd, which aggregates and stores them. We then extract the data into graphs on our dashboard, and use it extensively for Nagios alerting as well. There's a basic sample client included in this repository that we use for those purposes, though you're right, it just gets you raw numbers out of the box. We're planning on releasing more of both the "emitters" that gather data, as well as a major part of our graphing/dashboard interface "soon". And point well taken about making it more obvious how to get started and what you can use it for. I'll work on improving the documentation.
- scoot 14y ago
  Thank you, that's very helpful. Looking forward to the emitters and dashboard whenever they're ready - I suspect they will help drive adoption of Batsd and encourage development of additional emitters.
  - noahnoahnoah 14y ago
    Definitely. There's a teaser screenshot of Flyash (the big, reusable chunk of our dashboard) towards the bottom of http://37signals.com/svn/posts/3091-pssst-your-rails-application-has-a-secret-to-tell-you http://37signals.com/svn/posts/3091-pssst-your-rails-applica... (that post also details some of the major emitter components we use).
    - scoot 14y ago
      "457,739 different metrics in Flyash" Oh...kay... Flyash looks very nice from the screenshots, but I'm interested to learn how you solve the discoverability problem with that much data. (Too much of a good thing?)
    - mtalantikite 14y ago
      Looks great, can't wait to see it open sourced. I've been dealing with the clunkiness of statsd/data -> graphite -> graphene for a dashboard, and have more than a handful of times almost started writing exactly what it looks like you already have done. Any idea when we'll be able to use/contribute to Flyash?
      - noahnoahnoah 14y ago
        A couple of weeks, probably, depending on how much I feel like working on it. It was designed to be modular and easily extracted, but still needs some cleanup work and has a few nasty bugs I'd like to fix first.
- julian37 14y ago
  Could you explain briefly why you chose to write a replacement for statsd, rather than improve on it? What aspects of statsd were you not happy with? (I don't have a horse in this race, I haven't used statsd before -- but I am planning to deploy some sort of statistics gathering soon and I wonder why I would choose your implementation over Etsy's, apart from the obvious appeal of the 37signals brand.)

caiusdurling 14y ago

YEAAAAAAAAH!

uggedal 14y ago

Those interested in lower memory consumption and efficiency should take a look at https://github.com/armon/statsite https://github.com/armon/statsite another statsd "clone". Originally a Python implementation statsite is now rewritten in C.

taligent 14y ago
I wouldn't exactly call that a clone since it doesn't support key features of statsd (e.g. gauges) and has unique features of its own (e.g. key/value pairs). Looks brilliant otherwise.
- DavidAbrams 14y ago
  +1 for saying "key/value" instead of "key-value".
- armon 14y ago
  It is actually just a difference of naming. Gauges in statsd operate the same as kv data in statsite. Originally, both projects referred to them as kv, this seems to be a more recent change in statsd.

bri3d 14y ago

Similar, but backed by Graphite: https://github.com/mojodna/metricsd https://github.com/mojodna/metricsd It's awesome to see the community standardizing around a simple way of transporting measurements into a storage and analytics system.

richyzhang 14y ago

I would like to see Flyash you mentioned in the readme as soon as possible. For a monitor system, it is a crucial part.

nwmcsween 14y ago

Why should I use this over http://collectd.org http://collectd.org?

mattyb 14y ago
You don't use this over collectd; you can use BatsD with it. These tools are complementary. collectd is very good at collecting system-level metrics and sending them somewhere. Batsd is a system for receiving those metrics and storing them. (Note that collectd has a graphite plugin, but not one for (B|St)atsD. You'd need to write a proxy.)

igs123 14y ago

Content was really very informative. From http://www.indiagiftservices.com http://www.indiagiftservices.com