6 ms·
> Spark is for when you have a hundreds of machines worth of processing to do Absolutely agree. However, most uses of Spark I've seen in my career are people t
by moooo99 2y ago
> Spark is for when you have a hundreds of machines worth of processing to do
Absolutely agree. However, most uses of Spark I've seen in my career are people thinking they have hundreds of machines worth of processing to do.
- Marazan 2y agoAnd even when you jave quite a lot of machines worth of processing some single threaded streaming of data on a single machine can still beat out any distributed framework as the the overhead of distribution is large.
- 0cf8612b2e1e 2y agoA favorite paper, “Scalability! But at what COST?”. Authors show a single machine implementation (even single threaded) can wipe the floor with the maximum parallel capable implementation. http://dsrg.pdos.csail.mit.edu/2016/06/26/scalability-cost/ http://dsrg.pdos.csail.mit.edu/2016/06/26/scalability-cost/