3 ms·
To handle very high throughput, storing RRD files on a ramdisk works surprisingly well, if you can afford the cost and the loss of a few seconds of data - which
by shykes 15y ago
To handle very high throughput, storing RRD files on a ramdisk works surprisingly well, if you can afford the cost and the loss of a few seconds of data - which most of the time you can.
A simple tar + gzip is all you need to flush to disk, at the frequency of your choice. It turns out rrd write operations are safe enough to do this without corruption. And the IO cost is minimal compared to rrdcache: rrd data compresses extremely well.
- otterley 15y agoInteresting idea, but not a very efficient use of memory. Not only would you have to reserve memory for the ramdisk (assuming you even have enough to store all the files), but more precious memory would be wasted by buffer-caching the freshly-written tarred archives. You'd be sacrificing memory that would be otherwise used to service reads on frequently-accessed RRD files.
- shykes 15y agoYour point is technically valid but irrelevant in practice. The compressed tar is surprisingly small and you can stream it to S3 if you really want to shave off megabytes. But more importantly, you don't care. This will give you 3 orders of magnitude better write throughput with 2 hours of work. The savings in engineering time alone will buy you 1000x the ram you wasted! With that kind of achievement, my advice would be leave over-optimization for later and buy yourself a drink :)
- otterley 15y agoIt's not irrelevant in practice, at least not for me. I deal in enormous amounts of data that couldn't possibly fit into RAM.