4 ms·
The issues are a bit more fundamental. You can't practically fit 50TB on one machine and have reasonable performance, that means multiple machines with the dat
by bbrazil 12y ago
The issues are a bit more fundamental.
You can't practically fit 50TB on one machine and have reasonable performance, that means multiple machines with the data spread across them.
There's then two potential issues:
1) You're doing 1-to-1 joins across tables in a query, network latency may be an issue at high query rates
2) You're going 1-to-many or many-to-many joins across tables in a query, the resulting combinatorial explosion of data is too much to handle
You want to have your inner loops/joins as deep down in the stack as possible. If you can structure things so all the heavy lifting stays inside one rack/machine/NUMA node/ processor/core you'll be able to scale a good bit further further.
Designing things not to require joins, denormalising and putting it in a column store like Cassandra is also a good approach.