Apache Spark has a great 'lazy' story, and I recommend it as a compute supplement for pandas and numpy. I'm always fascinated by the lengths the authors go thro

Apache Spark has a great 'lazy' story, and I recommend it as a compute supplement for pandas and numpy. I'm always fascinated by the lengths the authors go through to optimise queries, from the cluster to one's machine. Also, the integration with Apache Arrow over the years has bore much fruit, to the extent that a pyspark user can cross over to Pandas with little serde cost. Fun story, about 18 months ago I was consulting for a telco. One of my tasks was to run a segmentation model on their subscriber base. They wanted me to use SAS in their cluster, but because I had 'brought my infra/laptop', I told them that I could do it with Python instead. I got the gist of it working on Pandas, but it was 'slow'. So they asked me if I was going to run it on a sample, but I said I'll port it to Spark in the evening. The following morning I presented them with results, and the porting to Spark took about 2 hours (I hadn't used SparkML before), and the thing took an hour to run on the whole population.