3 ms·
The most important consideration is the scale. A low scale system will not require an optimum solution, merely a correct one. So I would say, the system is 5 m
by zaroth 9y ago
The most important consideration is the scale. A low scale system will not require an optimum solution, merely a correct one.
So I would say, the system is 5 minute ingest of about 1 million metrics. This is spread out over a half dozen locations, each which currently records in their own silo.
And aggregate metric is calculated with a 5 minute lag, which reads all the just-written data points and aggregates into sum-totals which are themselves stored and cached in one place. This is another million metrics basically stored separate from the rest.
But it doesn't really change the character of the system. In the end I'm trying to;
Write batches of mostly numeric data
Queries over time against those numbers
Aggregate data over different blocks of time; 5 minute, hourly, daily, monthly
Store it efficiently
Ensure redundancy, integrity
Seems like a simple and common enough problem to have been reasonably "solved" for orders of magnitude higher scale than I'm operating at.
- alaties 9y agoFrom what I've seen, each solution has been pretty specific to the situation. I've rarely seen one tech stack prevail at low to reasonable scale. At reasonable scale, I've seen people get really far with pure graphite setups by utilizing tools like [carbon-c-relay](https://github.com/grobian/carbon-c-relay https://github.com/grobian/carbon-c-relay), [carbonate](https://github.com/graphite-project/carbonate https://github.com/graphite-project/carbonate), and high integrity filesystems like zfs underneath. It's a very hands on operation though. Things like growing the cluster are hard to do without downtime. If constant growth and uptime is a concern, something like openTSDB might be a great choice. The complexity of setting up a Hadoop + HBase cluster is a pretty big upfront cost, but man is this thing the cockroach of time series data stores. Adding storage is just growing the HBase cluster. Querying across years of data is pretty simple and straightforward. For the complexity involved, openTSDB is worth it.