3 ms·
If a single node contains the relevant partition(s) has all the data then it could be that the compute happens there and the result is then sent back no map red
by gigatexal 3y ago
If a single node contains the relevant partition(s) has all the data then it could be that the compute happens there and the result is then sent back no map reduce work needed.
- riku_iki 3y agoI envision it as many nodes store many partitions, and initial compute happens locally, and then results are re-partitioned if needed for the next step. Trivial case is predicate push down, when predicate is applied on storage backend and not compute node. I think ClickHouse is an example of such architecture, and they don't have dedicated compute nodes.
- gigatexal 3y agoYeah there's no perfect way to do this but there are advantages to being flexible and the use-case you mention is a subset of what I am describing. Imagine a workload that is write heavy and handles very few reads say 20:1 writes to reads. Then having many storage nodes could speed up the reads and if you have say a fix set of nodes allocating 20x more storage nodes than compute nodes might make sense. The whole thing is a science and I do like the flexibility.
- riku_iki 3y agothere are many nodes in both my description and ClickHouse.