3 ms·
The problem with joins and reason why most distributed databases don't support them is primarily that the local machinery is complicated to implement. Given lo
by mslot 11y ago
The problem with joins and reason why most distributed databases don't support them is primarily that the local machinery is complicated to implement.
Given local join machinery, a join between 2 distributed tables can be achieved by joining the partitions of the tables locally after ensuring that tables are co-partitioned and partitions are co-located. That is, if performing A JOIN B ON (A.X = B.Y), ensure that A is partitioned by X and B is partitioned by Y with the same partitioning scheme such that every partition in A has a corresponding partition in B with which it can be joined. A distributed join can then be performed (in parallel) by joining each pair of partitions and concatenating the results.
Ideally, tables are co-partitioned and partitions are co-located in advance. The article describes scenarios in which the tables need to be repartitioned during the query, which can cause a lot of network traffic. However, the amount of traffic is limited in practice since filters and projections can be applied before shuffling any data.
The great thing about implementing a distributed database on top of PostgreSQL is that the local machinery already exists, and it is very good.