3 ms·

Might be cool once PySpark integrates with Polars, but for now like many others I’m stuck with dropping into pandas for non-vectorized operations

• jvican 9mo ago
  Is there any plan for this?
  • datsci_est_2015 9mo ago
    Not that I’m aware of. The Spark ecosystem seems a little too “stable” to be putting effort into that kind of development. Edit: hah, based on the sibling comment, I stand corrected
  • devin-petersohn 9mo ago
    Funny enough, I actually just (2 weeks ago) added support for streaming from Pyspark to Polars/DuckDB/etc through Arrow PyCapsule. By streaming, I mean actually streaming, not collecting all data at once. It won't be released probably until May/June but it's there: https://github.com/apache/spark/commit/ecf179c3485ba8bac72afd9105892d9798d23f8f https://github.com/apache/spark/commit/ecf179c3485ba8bac72af...