4 ms·
How the relevant lesson from the original post was “Actually, you can scale pandas these days” eludes me.
by alextheparrot 5y ago
How the relevant lesson from the original post was “Actually, you can scale pandas these days” eludes me.
- dikei 5y agoSince the original post criticizes the behavior of Panda runtime where it loads everything into memory before processing, I think replacing the underlying Panda runtime with Spark runtime is very much a valid solution. You can keep using Panda code for exploratory work, then with a little effort move it to production with Spark runtime.
- monkeybutton 5y agoAs an avid user of pandas: That solution is just kicking the can down the road.