4 ms·

Could you elaborate on the negative hdd aspects of Graphite? I'm evaluating using it and would love to know more!

• vidarh 12y ago
  (the short-ish version: make sure you test it with a realistic number of metrics and updates coming in from a realistic number of servers; if it can keep up on hardware you're willing to throw at it, then by all means go for Graphite - it has tons of flexibility that makes it a convenient way to get started. Just expect it to need a fairly hefty amount of disk IO compared to the amount of data you feed it in; the interfaces to Graphite are simple enough and narrow enough you can replace it later if need be, once you know what parts of it you're using) Graphite essentially assumes that disk is cheap. Even fast disks. And either you're not collecting very big data sets, or you roll up to more rarely updated series very rapidly, or you have a hefty high performance RAID to put it on, or you distribute it over multiple servers. And for a lot of people it's probably true it'd be cheaper to just buy a suitable set of SSD's and doing RAID10 and/or spreading it out over multiple servers or something than spend time putting together something else. But if you start seeing yourself needing to scale up substantially, it may be worth thinking through exactly what your needs are and whether you can get away with less by giving up on some of what Graphite offers. And one outcome of the expectation of fast disk is simply that Whisper - the underlying storage code - makes some assumptions that leads to lots of unnecessary system calls; lots of seeks that turns out to be no-ops, and small reads and writes. All of them are common problems that people tend to be unaware of, but they were painfully obvious after some strace usage. Just fixing that would improve Graphite performance substantially even without ditching functionality, but I really did not feel like digging into unfamiliar Python code to get halfway there when I could just as quickly replace the whole thing. That assumption of being prepared to use a fast RAID just for some metrics for our dashboards and monitoring didn't sit well with me in general. And given that we basically just used it to aggregate a set of time series over relatively recent history and don't need most of Graphite's features, and that we don't particularly care if the occasional server crash takes out some of the most recent history with it on one server when we can cheaply run a second collecting most of the same data (and would've ended up with more than that with Graphite), it seemed like worth doing things differently. I basically replaced Graphite (the subset that we use of it) with this, in the course of a few days: * A tiny statsd implementation in Ruby, that flushes data to a Redis with disk snapshots turned off. It does this every 10 seconds. * An aggregation script that's a couple of hundred lines that uses key based filtering to aggregate all 10-second data for any five minute interval into a new time series, and again aggregate all five-minute interval data into hourly time series, and so on, and that moves all data older than a few hours to a disk backed CouchDB (could put it into "anything" really) * A tiny Sinatra app that provides a somewhat Graphite compatible JSON output (but with no attempt to support the Graphite query syntax or server side image rendering. * A slightly modified Graphene version to build our dashboard on. It does by no means do everything Graphite does, but what it does, it does without brutally hammering disk. It consume a tiny fraction of the resources Graphite used too.