5 ms·

Interesting. My team currently uses (abuses?) postgres for timeseries data. You mind ansswering some general questions about your experience with timescale? You

Interesting. My team currently uses (abuses?) postgres for timeseries data. You mind ansswering some general questions about your experience with timescale? You said 100 million sensor samples. What was the upload/download frequency? Our application is pushing hundreds of millions of rows across many different data sources every day. On top of that, we are also querying the shit out of this data to run models and we need VERY quick queries. like 10-100ms speed. How do you think timescaleDB would handle that size and also velocity of data?

merb 7y ago
actually the last time I checked/used it timescaledb did use parallel queries and had a good partinoning so that the parallel queries would even scale really good on a single node.
valyala 7y ago
TimescaleDB could fit your workload if PostgreSQL fits you. The main issue with PostgreSQL and TimescaleDB is big amounts of storage space required for huge time series data volumes. There are reports that storing data on ZSF can reduce the required storage space. Probably, ClickHouse [1] would fit better your needs. It can write millions of rows per second [2]. It can scan billions of rows per second on a single node and it scales to multiple nodes. Also I'd recommend taking a look at other open-source TSDBs with cluster support: - M3DB [3] - Cortex [4] - VictoriaMetrics [5] These TSDBs speak PromQL instead of SQL. PromQL is specially optimized query language for typical time series queries [6]. [1] https://clickhouse.yandex https://clickhouse.yandex [2] https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ https://blog.cloudflare.com/http-analytics-for-6m-requests-p... [3] https://www.m3db.io/ https://www.m3db.io/ [4] https://github.com/cortexproject/cortex https://github.com/cortexproject/cortex [5] https://github.com/VictoriaMetrics/VictoriaMetrics/ https://github.com/VictoriaMetrics/VictoriaMetrics/ [6] https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085 https://medium.com/@valyala/promql-tutorial-for-beginners-9a...
- akulkarni 7y ago
  We are actively working on native compression in TimescaleDB, with really promising early results. More to come.

merb 7y ago

actually the last time I checked/used it timescaledb did use parallel queries and had a good partinoning so that the parallel queries would even scale really good on a single node.

valyala 7y ago

TimescaleDB could fit your workload if PostgreSQL fits you. The main issue with PostgreSQL and TimescaleDB is big amounts of storage space required for huge time series data volumes. There are reports that storing data on ZSF can reduce the required storage space. Probably, ClickHouse [1] would fit better your needs. It can write millions of rows per second [2]. It can scan billions of rows per second on a single node and it scales to multiple nodes. Also I'd recommend taking a look at other open-source TSDBs with cluster support: - M3DB [3] - Cortex [4] - VictoriaMetrics [5] These TSDBs speak PromQL instead of SQL. PromQL is specially optimized query language for typical time series queries [6]. [1] https://clickhouse.yandex https://clickhouse.yandex [2] https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ https://blog.cloudflare.com/http-analytics-for-6m-requests-p... [3] https://www.m3db.io/ https://www.m3db.io/ [4] https://github.com/cortexproject/cortex https://github.com/cortexproject/cortex [5] https://github.com/VictoriaMetrics/VictoriaMetrics/ https://github.com/VictoriaMetrics/VictoriaMetrics/ [6] https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085 https://medium.com/@valyala/promql-tutorial-for-beginners-9a...

akulkarni 7y ago
We are actively working on native compression in TimescaleDB, with really promising early results. More to come.