3 ms·
My experience is that Polars generally isn't supported by third party libraries, hopefully tjst'll change soon but in the meantime it has some pretty snappy to
by benrutter 4y ago
My experience is that Polars generally isn't supported by third party libraries, hopefully tjst'll change soon but in the meantime it has some pretty snappy to and from pandas functionality, so converting when you hit a library that needs pandas often still brings a good deal of speedups.
- wenc 4y agoTo be honest I haven’t actually worked with any third party libraries that actually supported Pandas directly — most ML libs like Scikit require Numpy arrays (not Pandas dataframes) as inputs so I’ve always had to cast to Numpy from Pandas anyway. But yes, Polars and DuckDB can easily cast to Pandas and also read Pandas dataframes in memory. I have some legacy data transformations that are mostly DuckDB but involve some intermediate steps in Pandas (because I didn’t want to rewrite them) and it’s all seamless (though not zero copy as it would be in a pure Arrow workflow). And ironically DuckDB can query Pandas dataframes faster than Pandas itself due to its vectorized engine.