3 ms·

As lobster_johnson stated, Prometheus storage retention is a command line launch option. Prometheus is not intentionally designed as a long term cold storage o

As lobster_johnson stated, Prometheus storage retention is a command line launch option. Prometheus is not intentionally designed as a long term cold storage option for metrics. You _can_ store metrics for as long as your storage allows, but Prometheus is not going to replicate or manage that data to prevent long term degradation. Depending on your long term needs, the preferred pattern is to roll data off Prometheus into a metrics store that better handles data over the scope of months. Rolling data off of Prometheus is done with an exporter and documented [here](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage https://prometheus.io/docs/operating/integrations/#remote-en...). In most use operational use cases I've come across, we've only needed about a month of data, so keeping it in Prometheus was kosher. YMMV.

zaroth 9y ago
What if you want to keep time-series data for accounting purposes forever? Right now I use graphite but would love something that also handles replication/redundancy with a good query language & enough performance to also use it to fetch the underlying data used to render front-end graphs for users.
- alaties 9y ago
  The answer to these kinds of questions is very, very dependant on requirements and available resources. And there are rarely easy, out of the box answers. The big questions to ask are: - what data is of interest - what are the requested ingest patterns (how is data getting into this system? How frequently? What rate of ingest is expected?) - what are the requested query patterns (who is doing queries? What do those queries look like specifically? Are people querying over unbounded time ranges or are people doing more focused queries? Do queries regularly involve aggregations or not?) - what is the requested SLA (i.e. how much partial down time is okay? How much full downtime is okay? What kind of query response time do you need to target?) - what resources are/will be available (money, man power, compute, storage, tech on hand) It's possible that a time series data store might not be the correct system choice once these questions are answered. It's not unusual to see data split or copied into multiple systems to answer all requirements.
  - alaties 9y ago
    Feel free to reach out to me if you want to talk through it. alex@laties.info
  - zaroth 9y ago
    The most important consideration is the scale. A low scale system will not require an optimum solution, merely a correct one. So I would say, the system is 5 minute ingest of about 1 million metrics. This is spread out over a half dozen locations, each which currently records in their own silo. And aggregate metric is calculated with a 5 minute lag, which reads all the just-written data points and aggregates into sum-totals which are themselves stored and cached in one place. This is another million metrics basically stored separate from the rest. But it doesn't really change the character of the system. In the end I'm trying to; Write batches of mostly numeric data Queries over time against those numbers Aggregate data over different blocks of time; 5 minute, hourly, daily, monthly Store it efficiently Ensure redundancy, integrity Seems like a simple and common enough problem to have been reasonably "solved" for orders of magnitude higher scale than I'm operating at.
    - alaties 9y ago
      From what I've seen, each solution has been pretty specific to the situation. I've rarely seen one tech stack prevail at low to reasonable scale. At reasonable scale, I've seen people get really far with pure graphite setups by utilizing tools like [carbon-c-relay](https://github.com/grobian/carbon-c-relay https://github.com/grobian/carbon-c-relay), [carbonate](https://github.com/graphite-project/carbonate https://github.com/graphite-project/carbonate), and high integrity filesystems like zfs underneath. It's a very hands on operation though. Things like growing the cluster are hard to do without downtime. If constant growth and uptime is a concern, something like openTSDB might be a great choice. The complexity of setting up a Hadoop + HBase cluster is a pretty big upfront cost, but man is this thing the cockroach of time series data stores. Adding storage is just growing the HBase cluster. Querying across years of data is pretty simple and straightforward. For the complexity involved, openTSDB is worth it.
- gbrayut 9y ago
  We store 18-24 months of data in OpenTSDB with good performance and compression, but it can be a pain to manage if you don't have experience working with HBase. InfluxDB has commercial cluster support, and there are quite a few other options. See https://blog.outlyer.com/top10-open-source-time-series-databases https://blog.outlyer.com/top10-open-source-time-series-datab... for a good comparison list.
- bbrazil 9y ago
  > What if you want to keep time-series data for accounting purposes forever? Prometheus is not recommended for uses where 100% accuracy is required, see https://prometheus.io/docs/introduction/overview/#when-does-it-not-fit https://prometheus.io/docs/introduction/overview/#when-does-...? I'd also be wary of using Graphite for such a use case. For billing a more traditional database is probably best.

zaroth 9y ago

What if you want to keep time-series data for accounting purposes forever? Right now I use graphite but would love something that also handles replication/redundancy with a good query language & enough performance to also use it to fetch the underlying data used to render front-end graphs for users.

alaties 9y ago
The answer to these kinds of questions is very, very dependant on requirements and available resources. And there are rarely easy, out of the box answers. The big questions to ask are: - what data is of interest - what are the requested ingest patterns (how is data getting into this system? How frequently? What rate of ingest is expected?) - what are the requested query patterns (who is doing queries? What do those queries look like specifically? Are people querying over unbounded time ranges or are people doing more focused queries? Do queries regularly involve aggregations or not?) - what is the requested SLA (i.e. how much partial down time is okay? How much full downtime is okay? What kind of query response time do you need to target?) - what resources are/will be available (money, man power, compute, storage, tech on hand) It's possible that a time series data store might not be the correct system choice once these questions are answered. It's not unusual to see data split or copied into multiple systems to answer all requirements.
- alaties 9y ago
  Feel free to reach out to me if you want to talk through it. alex@laties.info
- zaroth 9y ago
  The most important consideration is the scale. A low scale system will not require an optimum solution, merely a correct one. So I would say, the system is 5 minute ingest of about 1 million metrics. This is spread out over a half dozen locations, each which currently records in their own silo. And aggregate metric is calculated with a 5 minute lag, which reads all the just-written data points and aggregates into sum-totals which are themselves stored and cached in one place. This is another million metrics basically stored separate from the rest. But it doesn't really change the character of the system. In the end I'm trying to; Write batches of mostly numeric data Queries over time against those numbers Aggregate data over different blocks of time; 5 minute, hourly, daily, monthly Store it efficiently Ensure redundancy, integrity Seems like a simple and common enough problem to have been reasonably "solved" for orders of magnitude higher scale than I'm operating at.
  - alaties 9y ago
    From what I've seen, each solution has been pretty specific to the situation. I've rarely seen one tech stack prevail at low to reasonable scale. At reasonable scale, I've seen people get really far with pure graphite setups by utilizing tools like [carbon-c-relay](https://github.com/grobian/carbon-c-relay https://github.com/grobian/carbon-c-relay), [carbonate](https://github.com/graphite-project/carbonate https://github.com/graphite-project/carbonate), and high integrity filesystems like zfs underneath. It's a very hands on operation though. Things like growing the cluster are hard to do without downtime. If constant growth and uptime is a concern, something like openTSDB might be a great choice. The complexity of setting up a Hadoop + HBase cluster is a pretty big upfront cost, but man is this thing the cockroach of time series data stores. Adding storage is just growing the HBase cluster. Querying across years of data is pretty simple and straightforward. For the complexity involved, openTSDB is worth it.
gbrayut 9y ago
We store 18-24 months of data in OpenTSDB with good performance and compression, but it can be a pain to manage if you don't have experience working with HBase. InfluxDB has commercial cluster support, and there are quite a few other options. See https://blog.outlyer.com/top10-open-source-time-series-databases https://blog.outlyer.com/top10-open-source-time-series-datab... for a good comparison list.
bbrazil 9y ago
> What if you want to keep time-series data for accounting purposes forever? Prometheus is not recommended for uses where 100% accuracy is required, see https://prometheus.io/docs/introduction/overview/#when-does-it-not-fit https://prometheus.io/docs/introduction/overview/#when-does-...? I'd also be wary of using Graphite for such a use case. For billing a more traditional database is probably best.