3 ms·
Every solution for robust joins sorting, hashing, or partitioning of some form and if you squint hard partitioning and hashing are very closely related to sorti
by timgarmstrong 5y ago
Every solution for robust joins sorting, hashing, or partitioning of some form and if you squint hard partitioning and hashing are very closely related to sorting, so I understand conceptually it seems samey - it kinda boils down to different schemes for partitioning the data.
The implementations and performance characteristics really aren't close at all.
The hash-based join starts in memory and degrades only under memory pressure, and, if it's a broadcast join, doesn't require data to be redistributed across nodes for each join. It also allows streaming one input through the join without sorting or otherwise buffering it in the join.
The old-style sort-merge join on map reduce requires both inputs to the join to be written to storage, shuffled across nodes and fully sorted even in the best case.
- trhway 5y ago>The old-style sort-merge join on map reduce requires both inputs to the join to be written to storage, shuffled across nodes and fully sorted even in the best case It isn't full sort of both inputs. It sorts only what is on given node.