4 ms·

Hi @jnordwick: I talk about this in another response and in the parent article. Individual time intervals are also spread across the cluster. So if you are c

by mfreed 7y ago

Hi @jnordwick: I talk about this in another response and in the parent article. Individual time intervals are also spread across the cluster. So if you are collecting data from a lot of, say, sensors, servers, or financial instruments, then reads/writes for the same time interval are then parallelized across servers. https://blog.timescale.com/blog/building-a-distributed-time-series-database-on-postgresql#architecture-access-nodes-and-data-nodes https://blog.timescale.com/blog/building-a-distributed-time-...

jnordwick 7y ago
I saw, my understanding is that you basically have a sort on device, then time: this helps some for individual queries devices to some extend (but prob hurts when you are inserts at 500 places instead of 1 or when you have queries that span too many instruments). Point wast (and the others's i think) was the you often have as very hot segment and yesterday's data is only used at night for example. and you can have a hot device (eg, top 10 symbols). the parting doesn't help a lot there until you can spread the time around and rejoin (netezza used to do something similar and it wasn't very good at it). Do you ever rebalance the partitions? getting you top 10 symbols accidentally stuck on the same partition would be painful especially without a way to control it. splaying the record column-wise helps in this, but i'm not sure if you are doing this.
- mfreed 7y ago
  It's not hierarchical as you describe. You don't first partition on device, then on time. They are done simultaneously -- see this older post for an illustration/comparison: https://blog.timescale.com/blog/time-series-data-postgresql-10-vs-timescaledb-816ee808bac5/#postgresql-10-partitioning https://blog.timescale.com/blog/time-series-data-postgresql-... So this architecture fully allows various striping or distribution options across time and space, even though the default might collocate chunks belonging to the same device on the same machine (at least since the last elasticity event).

by mfreed 7y ago

jnordwick 7y ago

I saw, my understanding is that you basically have a sort on device, then time: this helps some for individual queries devices to some extend (but prob hurts when you are inserts at 500 places instead of 1 or when you have queries that span too many instruments). Point wast (and the others's i think) was the you often have as very hot segment and yesterday's data is only used at night for example. and you can have a hot device (eg, top 10 symbols). the parting doesn't help a lot there until you can spread the time around and rejoin (netezza used to do something similar and it wasn't very good at it). Do you ever rebalance the partitions? getting you top 10 symbols accidentally stuck on the same partition would be painful especially without a way to control it. splaying the record column-wise helps in this, but i'm not sure if you are doing this.

mfreed 7y ago
It's not hierarchical as you describe. You don't first partition on device, then on time. They are done simultaneously -- see this older post for an illustration/comparison: https://blog.timescale.com/blog/time-series-data-postgresql-10-vs-timescaledb-816ee808bac5/#postgresql-10-partitioning https://blog.timescale.com/blog/time-series-data-postgresql-... So this architecture fully allows various striping or distribution options across time and space, even though the default might collocate chunks belonging to the same device on the same machine (at least since the last elasticity event).