3 ms·
I envision it as many nodes store many partitions, and initial compute happens locally, and then results are re-partitioned if needed for the next step. Trivia
by riku_iki 3y ago
I envision it as many nodes store many partitions, and initial compute happens locally, and then results are re-partitioned if needed for the next step.
Trivial case is predicate push down, when predicate is applied on storage backend and not compute node.
I think ClickHouse is an example of such architecture, and they don't have dedicated compute nodes.
- gigatexal 3y agoYeah there's no perfect way to do this but there are advantages to being flexible and the use-case you mention is a subset of what I am describing. Imagine a workload that is write heavy and handles very few reads say 20:1 writes to reads. Then having many storage nodes could speed up the reads and if you have say a fix set of nodes allocating 20x more storage nodes than compute nodes might make sense. The whole thing is a science and I do like the flexibility.
- riku_iki 3y agothere are many nodes in both my description and ClickHouse.