3 ms·
Great question! Depends on your workload size, if the compute time is less than time it takes to transfer the data back and forth, then it might be a bad idea
by teitoklien 3y ago
Great question!
Depends on your workload size, if the compute time is less than time it takes to transfer the data back and forth, then it might be a bad idea to use it indeed.
Typically these solutions should only be tested after maxing out vertical scaling, before applying for horizontal scaling.
Network is one hell of a destroyer when it comes to advantages gained from distributed computing.
- spratzt 3y agoI would go even further and argue that for the vast majority of organizations vertical scaling is all you are ever going to need.
- sammysidhu 3y agohorizontal scaling however provides you with more aggregate network bandwidth. Most enterprises run workloads that downloads data from a data lake (usually S3), which is usually the bottleneck. Having horizontal scaling here allows the query leverage much higher network than just having a large single machine.