4 ms·

I can only recommend TimescaleDB. It solves the right problems (storing timeseries) while not creating new ones (deployment, backup, hot failover) as it relies

I can only recommend TimescaleDB. It solves the right problems (storing timeseries) while not creating new ones (deployment, backup, hot failover) as it relies on Postgres to provide the underlying infrastructure. I stored 100 million sensor samples in TimeScale and had not issues with scaling on medium sized boxes, despite issuing complex time-series queries. As for the hosting option, currently sadly AWS doesn’t offer Timescale as part of RDS. There are two options: Azure offers Timescale now as part of their hosted Postgres. Or you go with aiven.io who can host you postgres with TimeScaleDB on all cloud providers (AWS, GCP, Azure, DO, ?) as a service, including replicas and backups. Overall, I’m very happy to see the Postgres ecosystem growing.

brootstrap 7y ago
Interesting. My team currently uses (abuses?) postgres for timeseries data. You mind ansswering some general questions about your experience with timescale? You said 100 million sensor samples. What was the upload/download frequency? Our application is pushing hundreds of millions of rows across many different data sources every day. On top of that, we are also querying the shit out of this data to run models and we need VERY quick queries. like 10-100ms speed. How do you think timescaleDB would handle that size and also velocity of data?
- merb 7y ago
  actually the last time I checked/used it timescaledb did use parallel queries and had a good partinoning so that the parallel queries would even scale really good on a single node.
- valyala 7y ago
  TimescaleDB could fit your workload if PostgreSQL fits you. The main issue with PostgreSQL and TimescaleDB is big amounts of storage space required for huge time series data volumes. There are reports that storing data on ZSF can reduce the required storage space. Probably, ClickHouse [1] would fit better your needs. It can write millions of rows per second [2]. It can scan billions of rows per second on a single node and it scales to multiple nodes. Also I'd recommend taking a look at other open-source TSDBs with cluster support: - M3DB [3] - Cortex [4] - VictoriaMetrics [5] These TSDBs speak PromQL instead of SQL. PromQL is specially optimized query language for typical time series queries [6]. [1] https://clickhouse.yandex https://clickhouse.yandex [2] https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ https://blog.cloudflare.com/http-analytics-for-6m-requests-p... [3] https://www.m3db.io/ https://www.m3db.io/ [4] https://github.com/cortexproject/cortex https://github.com/cortexproject/cortex [5] https://github.com/VictoriaMetrics/VictoriaMetrics/ https://github.com/VictoriaMetrics/VictoriaMetrics/ [6] https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085 https://medium.com/@valyala/promql-tutorial-for-beginners-9a...
  - akulkarni 7y ago
    We are actively working on native compression in TimescaleDB, with really promising early results. More to come.
laughinghan 7y ago
DigitalOcean Managed Databases also comes with TimescaleDB built-in: https://www.digitalocean.com/docs/databases/postgresql/resources/supported-extensions/ https://www.digitalocean.com/docs/databases/postgresql/resou...
akulkarni 7y ago
If you are looking for hosted TimescaleDB, eg on AWS, you may want to check out Timescale Cloud [1], which we launched a couple months ago. Fully-managed TimescaleDB, including community and enterprise capabilities, high-availability, etc, available on AWS, GCP, and Azure. There are other options as well (Azure, DigitalOcean, Alibaba, Aiven) but they only offer the OSS version of TimescaleDB. More here (including a pricing calculator): https://www.timescale.com/cloud https://www.timescale.com/cloud [1] https://blog.timescale.com/blog/timescale-cloud-first-fully-managed-time-series-database-service-runs-on-aws-gcp-azure/ https://blog.timescale.com/blog/timescale-cloud-first-fully-...
GordonS 7y ago
I was going to post a very similar comment :) For on-prem deployments I'm using TimescaleDB on a single node with up to 100 million events, and for SaaS I'm using it on Azure, and the performance is little short of amazing! Something I particularly like is the almost instantaneous deletes (because of the "chunking" model) - perfect for data retention jobs that delete old data.
stalller 7y ago
Thanks for the recommendation! And glad to hear TimescaleDB meets your needs. I wanted to add some color to hosting / deployment options around TimescaleDB. Options are as follows: [1] Fully-Managed TimescaleDB Enterprise on Timescale Cloud TimescaleDB Open Source hosted on Public Clouds: [2] Azure PostgreSQL [3] DO Managed-Postgres (mentioned above) [4] Alibaba Cloud Not hosted, but managed service options: [5] ClusterControl from our friends at Severalnines --- [1]https://www.timescale.com/cloud https://www.timescale.com/cloud [2]https://azure.microsoft.com/en-us/blog/power-iot-and-time-series-workloads-with-timescaledb-for-azure-database-for-postgresql/ https://azure.microsoft.com/en-us/blog/power-iot-and-time-se... [3]https://www.digitalocean.com/docs/databases/postgresql/resources/supported-extensions/ https://www.digitalocean.com/docs/databases/postgresql/resou... [4]https://www.alibabacloud.com/blog/sql-and-timescaledb_595169 https://www.alibabacloud.com/blog/sql-and-timescaledb_595169 [5]https://severalnines.com/blog/advanced-database-monitoring-management-timescaledb https://severalnines.com/blog/advanced-database-monitoring-m...
keithnoizu 7y ago
I use RiakTS although unfortunately with Basho's demise it's future is in a bit of jeopardy.

brootstrap 7y ago

Interesting. My team currently uses (abuses?) postgres for timeseries data. You mind ansswering some general questions about your experience with timescale? You said 100 million sensor samples. What was the upload/download frequency? Our application is pushing hundreds of millions of rows across many different data sources every day. On top of that, we are also querying the shit out of this data to run models and we need VERY quick queries. like 10-100ms speed. How do you think timescaleDB would handle that size and also velocity of data?

merb 7y ago
actually the last time I checked/used it timescaledb did use parallel queries and had a good partinoning so that the parallel queries would even scale really good on a single node.
valyala 7y ago
TimescaleDB could fit your workload if PostgreSQL fits you. The main issue with PostgreSQL and TimescaleDB is big amounts of storage space required for huge time series data volumes. There are reports that storing data on ZSF can reduce the required storage space. Probably, ClickHouse [1] would fit better your needs. It can write millions of rows per second [2]. It can scan billions of rows per second on a single node and it scales to multiple nodes. Also I'd recommend taking a look at other open-source TSDBs with cluster support: - M3DB [3] - Cortex [4] - VictoriaMetrics [5] These TSDBs speak PromQL instead of SQL. PromQL is specially optimized query language for typical time series queries [6]. [1] https://clickhouse.yandex https://clickhouse.yandex [2] https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ https://blog.cloudflare.com/http-analytics-for-6m-requests-p... [3] https://www.m3db.io/ https://www.m3db.io/ [4] https://github.com/cortexproject/cortex https://github.com/cortexproject/cortex [5] https://github.com/VictoriaMetrics/VictoriaMetrics/ https://github.com/VictoriaMetrics/VictoriaMetrics/ [6] https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085 https://medium.com/@valyala/promql-tutorial-for-beginners-9a...
- akulkarni 7y ago
  We are actively working on native compression in TimescaleDB, with really promising early results. More to come.

laughinghan 7y ago

DigitalOcean Managed Databases also comes with TimescaleDB built-in: https://www.digitalocean.com/docs/databases/postgresql/resources/supported-extensions/ https://www.digitalocean.com/docs/databases/postgresql/resou...

akulkarni 7y ago

If you are looking for hosted TimescaleDB, eg on AWS, you may want to check out Timescale Cloud [1], which we launched a couple months ago. Fully-managed TimescaleDB, including community and enterprise capabilities, high-availability, etc, available on AWS, GCP, and Azure. There are other options as well (Azure, DigitalOcean, Alibaba, Aiven) but they only offer the OSS version of TimescaleDB. More here (including a pricing calculator): https://www.timescale.com/cloud https://www.timescale.com/cloud [1] https://blog.timescale.com/blog/timescale-cloud-first-fully-managed-time-series-database-service-runs-on-aws-gcp-azure/ https://blog.timescale.com/blog/timescale-cloud-first-fully-...

GordonS 7y ago

I was going to post a very similar comment :) For on-prem deployments I'm using TimescaleDB on a single node with up to 100 million events, and for SaaS I'm using it on Azure, and the performance is little short of amazing! Something I particularly like is the almost instantaneous deletes (because of the "chunking" model) - perfect for data retention jobs that delete old data.

stalller 7y ago

Thanks for the recommendation! And glad to hear TimescaleDB meets your needs. I wanted to add some color to hosting / deployment options around TimescaleDB. Options are as follows: [1] Fully-Managed TimescaleDB Enterprise on Timescale Cloud TimescaleDB Open Source hosted on Public Clouds: [2] Azure PostgreSQL [3] DO Managed-Postgres (mentioned above) [4] Alibaba Cloud Not hosted, but managed service options: [5] ClusterControl from our friends at Severalnines --- [1]https://www.timescale.com/cloud https://www.timescale.com/cloud [2]https://azure.microsoft.com/en-us/blog/power-iot-and-time-series-workloads-with-timescaledb-for-azure-database-for-postgresql/ https://azure.microsoft.com/en-us/blog/power-iot-and-time-se... [3]https://www.digitalocean.com/docs/databases/postgresql/resources/supported-extensions/ https://www.digitalocean.com/docs/databases/postgresql/resou... [4]https://www.alibabacloud.com/blog/sql-and-timescaledb_595169 https://www.alibabacloud.com/blog/sql-and-timescaledb_595169 [5]https://severalnines.com/blog/advanced-database-monitoring-management-timescaledb https://severalnines.com/blog/advanced-database-monitoring-m...

keithnoizu 7y ago

I use RiakTS although unfortunately with Basho's demise it's future is in a bit of jeopardy.