3 ms·

As of a few days ago (Apache Spark 3.2 release), you can use the pandas API on a Spark cluster: https://databricks.com/blog/2021/10/04/pandas-api-on-upcoming-ap

• alextheparrot 5y ago
  How the relevant lesson from the original post was “Actually, you can scale pandas these days” eludes me.
  • dikei 5y ago
    Since the original post criticizes the behavior of Panda runtime where it loads everything into memory before processing, I think replacing the underlying Panda runtime with Spark runtime is very much a valid solution. You can keep using Panda code for exploratory work, then with a little effort move it to production with Spark runtime.
    • monkeybutton 5y ago
      As an avid user of pandas: That solution is just kicking the can down the road.
• ldng 5y ago
  And now you have a whole set of new problems ?