4 ms·

I was actually looking at a bunch of open source time series databases and settled on kairosdb but this looks pretty nice. I think there is a hackernews rule s

I was actually looking at a bunch of open source time series databases and settled on kairosdb but this looks pretty nice. I think there is a hackernews rule someplace that a more interesting tech alternative shows up right when you decided to go with something else. For reference here is the list I created when researching these: http://opentsdb.net/overview.html http://opentsdb.net/overview.html Built on HBASE http://www.gocircuit.org/vena.html http://www.gocircuit.org/vena.html Built on go + go'circult uses google's leveldb https://code.google.com/p/kairosdb/ https://code.google.com/p/kairosdb/ A rewrite of opentsdb which can use Cassandra http://blueflood.io/ http://blueflood.io/ Built by Rackspace, decent but still seems a bit immature http://graphite.wikidot.com/ http://graphite.wikidot.com/ Obligatory Graphite reference (uses whisper, new backend called 'ceres' is being developed) https://github.com/agoragames/kairos https://github.com/agoragames/kairos (yet another 'kairos', alternative backends for graphite - SQL, redis, or mongo) Riak seems to be popular with SASS metric providers (hosted graphite, boundary). There isn’t any code but there are a couple of talks that explain how and why they went with Riak: http://basho.com/hosted-graphite-uses-riak-to-store-all-customer-metrics/ http://basho.com/hosted-graphite-uses-riak-to-store-all-cust... http://boundary.com/blog/tag/tech-talks/ http://boundary.com/blog/tag/tech-talks/ http://vimeo.com/42902962 http://vimeo.com/42902962

vosper 13y ago
Thanks for sharing all these links! I'm always trying to find new and interesting DB technologies, especially time-series. Can you share anything of your experiences with kairosdb so far - what's the use case and how has it performed?
- hoopism 13y ago
  Thanks! We also went through a similar assessment and came away with kairosdb being the favorable option. Have not started any integration so early thoughts would be helpful. We def favored cassandra over HBASE and the little work we did with opentsdb went well.
jwatte 13y ago
You didn't look at istatd? It does 150000 counters at three different retention intervals every ten seconds, with average, total, standard deviation, mom and max for all metrics. We looked at other options (including open tsdb and graphite) before building this. Https://github.com/imvu-open/istatd
cnlwsu 13y ago
Actually, as a person who used most of these and built things similarly this project seems very interesting and useful and tackles a lot of the annoyances of those above. Kinda a combination of otsdb and esper but without a big requirement of setting up other technologies (I don't have to become or hire a HBase expert to maintain it). Having a bunch of online algorithms executed against the data is exciting too from a perspective of future improvements alsom I would love to see anomaly detection built in.
JulianMorrison 13y ago
https://github.com/imvu-open/istatd/wiki https://github.com/imvu-open/istatd/wiki is my favouite. C++, compile to a binary and then it can send updates in a tree of agents to a master node, and maintain a replica of the master, and it does graphs too, everything you need in one.
philjackson 13y ago
Rolling your own: http://blog.apiaxle.com/post/storing-near-realtime-stats-in-redis/ http://blog.apiaxle.com/post/storing-near-realtime-stats-in-...
- e12e 13y ago
  I should probably look at the code, but reading the post it isn't entirely clear to me: are you updating all three (second, minute, hour) keys on every insert? And are you just using the "last value" as the value for the key (so the value for minute becomes the last value of second that falls in that interval? I need to read up on rrdtool as well, but I wonder if it would make much difference (good or bad) to store the mean or other average as the "higher up value" (ie: the average of the past 60 seconds as the minute value) ?

vosper 13y ago

Thanks for sharing all these links! I'm always trying to find new and interesting DB technologies, especially time-series. Can you share anything of your experiences with kairosdb so far - what's the use case and how has it performed?

hoopism 13y ago
Thanks! We also went through a similar assessment and came away with kairosdb being the favorable option. Have not started any integration so early thoughts would be helpful. We def favored cassandra over HBASE and the little work we did with opentsdb went well.

jwatte 13y ago

You didn't look at istatd? It does 150000 counters at three different retention intervals every ten seconds, with average, total, standard deviation, mom and max for all metrics. We looked at other options (including open tsdb and graphite) before building this. Https://github.com/imvu-open/istatd

cnlwsu 13y ago

Actually, as a person who used most of these and built things similarly this project seems very interesting and useful and tackles a lot of the annoyances of those above. Kinda a combination of otsdb and esper but without a big requirement of setting up other technologies (I don't have to become or hire a HBase expert to maintain it). Having a bunch of online algorithms executed against the data is exciting too from a perspective of future improvements alsom I would love to see anomaly detection built in.

JulianMorrison 13y ago

https://github.com/imvu-open/istatd/wiki https://github.com/imvu-open/istatd/wiki is my favouite. C++, compile to a binary and then it can send updates in a tree of agents to a master node, and maintain a replica of the master, and it does graphs too, everything you need in one.

philjackson 13y ago

Rolling your own: http://blog.apiaxle.com/post/storing-near-realtime-stats-in-redis/ http://blog.apiaxle.com/post/storing-near-realtime-stats-in-...

e12e 13y ago
I should probably look at the code, but reading the post it isn't entirely clear to me: are you updating all three (second, minute, hour) keys on every insert? And are you just using the "last value" as the value for the key (so the value for minute becomes the last value of second that falls in that interval? I need to read up on rrdtool as well, but I wonder if it would make much difference (good or bad) to store the mean or other average as the "higher up value" (ie: the average of the past 60 seconds as the minute value) ?