3 ms·
In a twist, pandas programs don't get parallelized on Spark. Someone had to go and write a parallel layer that duplicated the pandas API, because otherwise you
by legerdemain 4y ago
In a twist, pandas programs don't get parallelized on Spark. Someone had to go and write a parallel layer that duplicated the pandas API, because otherwise you ended up with the entire pandas program executing on a single executor.
- alexott 4y agothere is Pandas on Spark, included into Spark itself (originally Koalas) - the switch to it is very easy, and you get parallelization.