3 ms·
Yes but most data-heavy tasks are parallelizable. SQL itself is naturally parallelizable. There's a reason Apache RAPIDs, Voltron, Kinetica, Sqream, etc exist.
by dangoldin 2y ago
Yes but most data-heavy tasks are parallelizable. SQL itself is naturally parallelizable. There's a reason Apache RAPIDs, Voltron, Kinetica, Sqream, etc exist.
Full transparency I don't have huge amount of experience at working on this massive scale and to your point you need to understand the problem and constraints before you propose a solution.
- cgio 2y agoYou have to revisit the assertion that SQL is naturally paralleliseable. As a guide have a look at the semantics around Spark shuffles.
- llm_trw 2y agoThere are more asterisks attached to each assertion you're making than you can shake a stick at. There is always a 'simple' transformation that the business requires which turns out to need n^2 space that kills the server it's running on because people believe everything you said above. Or in other words: most of the time you don't need a seat belt in a car either.