4 ms·
This was a great article, and I'm a competitor! :) Learned a lot about the history. My one nit-pick is they say distributed joins are hard though. But they don
by marknadal 8y ago
This was a great article, and I'm a competitor! :) Learned a lot about the history.
My one nit-pick is they say distributed joins are hard though. But they don't really justify it other than the assumption other engineers have come from a traditional DB background. It is not a hard problem if you try to solve it on your own though: You do the join on the client, incrementally (or debounced if necessary) as the data is streamed. This works very well at scale in production (TB/day) for our system.
- politician 8y agoThere are a variety of distributed join scheduling models to consider depending on the latency of the network and the size of the data sets. Techniques like Bloom filters used to accelerate joins by minimizing the data transmitted are novel to many developers. Layer in HA and transactions, and it becomes hard.