4 ms·
Yes, we're working on a distributed version of Timescale as you describe. But two clarifications: 1. It can aggregate better than you might think. We've had
by mfreed 9y ago
Yes, we're working on a distributed version of Timescale as you describe.
But two clarifications:
1. It can aggregate better than you might think. We've had people run single-node Timescale with 20+ disks, then couple that with query parallelization, and you can do pretty good aggregation over larger datasets.
Plus because the way the data is partitioned, a GROUPBY will actually get good localization over the disjoint data (i.e., groups can be local to a chunk) and generate more efficient plans given the smaller per-chunk indexes.
(And the various cloud platforms make it really easy to attach many disks to a single machine. Our our published benchmarking is on network-attached SSDs.)
2. You can use read-only clustering today, i.e., with standard Postgres synchronous or asynchronous replication. So you can scale your query rates with the replicas as well.
- buremba 9y agoThanks for the clarification. 1. Do you use Postgresql 9.6 query parallelization (https://www.postgresql.org/docs/9.6/static/parallel-plans.html#PARALLEL-AGGREGATION https://www.postgresql.org/docs/9.6/static/parallel-plans.ht...) or your own method for processing chunks parallelly? When we have >1B rows with >20 columns, the IO usually becomes a huge the bottleneck in our experience. If you use multiple disks and parallelize the work among different CPU cores, it would help I guess.
- mfreed 9y agoCurrently support 9.6 query parallelization. Also considering extending with some of our own methods on chunks as well. (Timescale supports multiple disks either through RAID or tablespaces. Unlike PG, you can add multiple tablespaces to a single hypertable.) Happy to also go into more details on Slack (https://slack-login.timescale.com https://slack-login.timescale.com) or email.