6 ms·
Brubeck – A statsd-compatible metrics aggregator
- moogly 11y agoPerchance named after Dave Brubeck?
- jssjr 11y agoIndeed!
- robhanlon 11y agoComplete with a subtle joke about odd meter!
- agumonkey 11y agoWaiting for a prng named Ornette.
- unhammer 11y agoAre you sure? My middle name is Brubeck as well, and there is some overlap in features: * Brubeck only runs on Linux. It won't even build on Mac OS X. … Brubeck has the following dependencies: * A Turing-complete computing device running a modern version of the Linux kernel … The are several ways to interact with a running Brubeck daemon. Though I guess that jazz guy is more famous.
- jmsdnns 11y agoI once built a web framework named Brubeck (https://github.com/j2labs/brubeck https://github.com/j2labs/brubeck). I stopped developing it a while ago, though. I am really happy to see another project, one that will probably reach more people than my web framework, honor the same wonderful musician I was trying to honor.
- zallarak 11y agoI came here to mention this! I saw you present this at the Recurse Center a long time ago.
- jmsdnns 11y agoYep! I built a significant amount of it while I was at Recurse Center. Proud to say that it all happened in RC's inaugural batch too. :)
- dberg 11y agoFunny i think i remember going to a brubeck meetup here in NY once. My old co-woker Patrick mentioned it and we ended up seeing a talk on it.
- jmsdnns 11y agoThat was me. BrubeckNYC. :)
- mcfunley 11y agoI was at Etsy when we made statsd, and I'm currently working on fixing the same problems at Stripe. This is super interesting and relevant to me personally, thanks for writing it all down! There are some details of how Etsy uses statsd that are not well-communicated. Etsy samples metrics aggressively to limit the amount of total traffic. And they monitor the packet error rates on the statsd boxes like hawks to keep the loss rate in check. Back when I was working there, if you added a high-volume counter without sampling it, the alarms would sound and you'd have an ops person tapping your shoulder pretty quickly. If you use statsd and skip either of these steps, the 40% loss that github experienced is what you get. AIUI Etsy's moved to a consistent hashing scheme that's at least vaguely similar to this. Node was not then, nor is now Etsy's area of expertise. We were going through an adolescent "let's just use every language" phase when we built statsd. I think the problems outlined here are solid supporting evidence that you should use a smallish set of tools and master them (a point of view which is very on-brand for Etsy engineering as it exists today).
- bbrazil 11y agoHave you considered pushing the aggregation into the applications, rather than doing it across the network? Having whatever library is sending data to statsd, instead keep counters/gauges in memory and then expose that on a regular basis would greatly reduce the data volumes involved as it's O(timeseries*frequency) rather than O(events). This is the approach we take with Prometheus, and based on the statsd setups of some people who've come talking to us there's scope for a reduction in network load of at least an order of magnitude without having to do any downsampling.
- mcfunley 11y agoYeah this has come up and it's reasonable, although it's a tricky/laborious migration in practice given a wide variety of things emitting stats. The statsd design choices here are mostly explained by the fact that Etsy uses it to collect from PHP. PHP doesn't afford a great way to aggregate in the client. (These are design choices that serve PHP well systemically, although it's limiting here.)
- jameskilton 11y agoThis is awesome, and more choice is definitely good, but I'm curious if Github tried out statsite[1] at all in this process? My guess is that these two tools were built at the exact same time solving the same problems. [1] https://github.com/armon/statsite https://github.com/armon/statsite
- jssjr 11y agoWe did not try out statsite, but it also looks great! Brubeck has been under development internally since October 2012. We're really excited to be able to share it with everyone.
- lpgauth 11y agostatsite is also based on a event-loop and is limited to a single core (no smp).
- touchingvirus 11y agoThis. We recently had to upgrade to a c4 instance in AWS because the single core on a c3 couldn't take what we were sending. We needed something like Brubeck because eventually the c4 will run out of headroom too.
- armon 11y agoTo clarify, statsite does spawn threads to do some of the aggregation and flushing at the end of the collection interval, but yes it is possible to saturate the main loop at a very high ingest rate.
- dpkp 11y agoswitched from statsd to statsite a few years ago and haven't looked back. highly recommend. same high quality code you'd expect from armon and the hashicorp crew (though statsite predates hashicorp, I believe).
- marceldegraaf 11y agoLooks great! Question for the author(s) of Brubeck: are you planning to add an InfluxDB backend, or is that something you'd want the community to pick up?
- rodionos 11y ago>A Turing-complete computing device running a modern version of the Linux kernel... I liked that! Any plans for pluggable backends in addition to BRUBECK_BACKEND_CARBON? Pls consider making it agnostic, i.e. using a wire protocol.
- leetrout 11y agoI don't really "know" golang but this seems like an area where it would have been useful. Am I think about this the right way? Would channels have helped handle some of the load effectively? Any gophers care to comment? :D
- Zariel 11y agoI know bitly have an implementation of statsd in golang [0]. Ive not used it in production but have contributed to it. Go is fast to a point, it is not as fast as C but it is easier to write fast Go than fast C. [0] https://github.com/bitly/statsdaemon https://github.com/bitly/statsdaemon
- sagichmal 11y agoChoosing C for this project is definitely... strange. Keith Rarick calls it "downright irresponsible" [1] — I'm not sure I disagree. [1] https://twitter.com/krarick/status/610502413007503360 https://twitter.com/krarick/status/610502413007503360
- jeremyjh 11y agoDoes he think there were no Garbage Collected languages available in 2007 when he started beanstalkd ? The arrogance is breathtaking.
- brazzledazzle 11y agoHe actually replied to a tweet that brought that up: "I was young(er) and stupid(er) then.."
- DasIch 11y agoWhat is the alternative? Rust isn't there yet when it comes to IO and there are certainly no other languages that qualify and allow a similar level of control.
- sagichmal 11y ago
- piotrp 11y agoThere already exist many statsd implementations from which two in C and four in golang. Why the need for another one? https://github.com/etsy/statsd/wiki#server-implementations https://github.com/etsy/statsd/wiki#server-implementations
- simonw 11y agoPresumably because they didn't exist (or weren't production-ready) three years ago: "After three years running in our production servers (with very good results both in performance and reliability), today we're finally releasing Brubeck as open-source."
- anton_gogolev 11y agoCan't help but plug our MIT-licensed port of Graphite/StatsD to .NET: [1]. [1]: https://bitbucket.org/aeroclub-it/statsify https://bitbucket.org/aeroclub-it/statsify
- deleted 11y ago[deleted]
- whost49 11y agoIs there a reason why you didn't just use collectd, which is written in C and has lots of plugins, including statsd, InfluxDB, Graphite, etc.?