3 ms·
Ask HN: Replacement backend for RRD
I am currently re-engineering an stats tracking, analysis, and alerting application which is based upon rrdtool. There are currently roughly 20k data points being monitored every minute and pushed to rrdcached. I'm seeing a substantial performance impact as the quantity of data points grow, even though I am running it on a fairly big box, as the initial implementation of our app targeted roughly 1000 data points.
Ultimately I'd like to not stress about scaling the application to 100k data points. I've thought of using Redis, PostgreSQL and Hadoop/HBase, all which are tackling different domain problems. One of my concerns is if I remove the lossy trending of rrd, the size of the data footprint is substantial at 20k data points (20k * 1440 * data lifetime * per-row-overhead).
Do you have any suggestions on a data backend, preferably something lossy like rrd?
- delano 16y agoI've worked a lot with Redis over the past 6 months (my own projects + analytics for yellowpages.ca) and not so much with rrd. Would you be interested in an email exchange?
- pedoh 16y agoAt a previous job we were using Ganglia (http://ganglia.sourceforge.net/ http://ganglia.sourceforge.net/) to collect both system metrics and custom metrics from our servers. At my current job, we still use Ganglia for redundant storage of our metrics, but then we feed the data into Graphite (http://graphite.wikidot.com/ http://graphite.wikidot.com/) for graphing. Graphite uses Whisper for it's storage mechanism. It's very similar to RRD. The author wrote a page about why he decided to write his own instead of leveraging RRD (http://graphite.wikidot.com/whisper http://graphite.wikidot.com/whisper). We've got over 60k metrics in our system; happy to talk shop if you want more info.