3 ms·
Use Round Robin Databases-- they're made for high volume timeseries data. They're fixed in size, so they won't grow after you create them. They also support au
by caw 15y ago
Use Round Robin Databases-- they're made for high volume timeseries data. They're fixed in size, so they won't grow after you create them. They also support automatic rollup, so you could save 1 datapoint per second for 10 minutes, then save the average of 15 datapoints for the past day, etc.
Caveats:
1) You can't change the schema once you create the file. Adding new fields means adding a new file or recreating your existing file
2) With heavy I/O you may need to put these files on a RAM disk, and then periodically flush the persistency.
- haarts 15y agoThat sounds very interesting. I could only find RRDTool which seems like a combination of a RRD and a graphing tool. Further some vague references about RRD in PostgreSQL. Am I missing something?
- caw 15y agoRRDTool is what you're looking for. The graphing portion is because you probably want to display the data. "rrdtool dump" will export to XML. You can use the RRD files without using the graphing utility. For an example, see the Cacti project.
- scraplab 15y agoThere's also Whisper, a Python library that's similar to RRD, and what Graphite uses for storage. http://graphite.wikidot.com/whisper http://graphite.wikidot.com/whisper