3 ms·

This sounds awesome! But is it the right approach if I am just running a simple Prometheus instance on my home NAS? I've wondered for a while how I can persist

by dom96 5y ago

This sounds awesome! But is it the right approach if I am just running a simple Prometheus instance on my home NAS? I've wondered for a while how I can persist my Prometheus timeseries, I guess I could use promscale for this, but maybe it's overkill for something this simple. Advice appreciated :)

derefr 5y ago
Indefinite persistence of time-series data "as-is", is a somewhat different use-case from putting them in a data warehouse so you can efficiently do rollups to them. Timescale seems to be useful for the latter, but I'm not sure it offers too much value for the former. I believe the state-of-the-art for plain-old Prometheus data retention is https://thanos.io/ https://thanos.io/ — my understanding is that it's a Prometheus remote storage integration (https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations https://prometheus.io/docs/prometheus/latest/storage/#remote...) that archives time-series data from Prometheus into an object-store, and then fetches/streams chunks back out from said object-store to serve requests. You could use it locally on your NAS, by running a Minio instance on there. But IMHO there wouldn't be much point in doing that, over just keeping all the data in Prometheus's own internal storage.
- cevian 5y ago
  (NB: Post author and Promscale dev here) Promscale does both data storage and analysis/rollups. It's like Thanos in that you can use it as a remote storage backend. It has the additional functionality of then aggregating/analyzing the data in SQL.
  - cyberpunk 5y ago
    Just be careful with Thanos if you're storing in s3. We got a 17k bill for rolling up/compacting our metrics which far outweighed the cost of just keeping them at the orig resolution.. :}
- dom96 5y ago
  > But IMHO there wouldn't be much point in doing that, over just keeping all the data in Prometheus's own internal storage. So far from what I've read Prometheus isn't built for that. It stores all its data in RAM and asking it to store any more leads to running out of memory quickly. What have I missed?

by dom96 5y ago

derefr 5y ago

Indefinite persistence of time-series data "as-is", is a somewhat different use-case from putting them in a data warehouse so you can efficiently do rollups to them. Timescale seems to be useful for the latter, but I'm not sure it offers too much value for the former. I believe the state-of-the-art for plain-old Prometheus data retention is https://thanos.io/ https://thanos.io/ — my understanding is that it's a Prometheus remote storage integration (https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations https://prometheus.io/docs/prometheus/latest/storage/#remote...) that archives time-series data from Prometheus into an object-store, and then fetches/streams chunks back out from said object-store to serve requests. You could use it locally on your NAS, by running a Minio instance on there. But IMHO there wouldn't be much point in doing that, over just keeping all the data in Prometheus's own internal storage.

cevian 5y ago
(NB: Post author and Promscale dev here) Promscale does both data storage and analysis/rollups. It's like Thanos in that you can use it as a remote storage backend. It has the additional functionality of then aggregating/analyzing the data in SQL.
- cyberpunk 5y ago
  Just be careful with Thanos if you're storing in s3. We got a 17k bill for rolling up/compacting our metrics which far outweighed the cost of just keeping them at the orig resolution.. :}
dom96 5y ago
> But IMHO there wouldn't be much point in doing that, over just keeping all the data in Prometheus's own internal storage. So far from what I've read Prometheus isn't built for that. It stores all its data in RAM and asking it to store any more leads to running out of memory quickly. What have I missed?