3 ms·
I find myself using Pandas .apply() and pd.read_csv() quite often which can get a little slow when dealing with lots of rows and files. Would you happen to kno
by dr_kiszonka 4y ago
I find myself using Pandas .apply() and pd.read_csv() quite often which can get a little slow when dealing with lots of rows and files.
Would you happen to know if these two functions are faster in Polars?
- raihansaputra 4y agoAnother alternative is to try Modin. They wrap a scalable engine (doesn't have to be multiple machine, speedup happens locally from single -> multithread) with Pandas API so you just need to change your imports. .apply() is also automatically parallelized through Modin. https://modin.readthedocs.io/en/stable/getting_started/quickstart.html https://modin.readthedocs.io/en/stable/getting_started/quick...
- dr_kiszonka 4y agoThis looks great! Thank you, raihansaputra!
- wenc 4y agoRead_csv - yes. I just tried loading a 5MM row csv. Pandas took 5.5 secs. Polars took 0.6 secs. .apply - might be faster in Polars but will not be as fast as using native expressions. That’s because applying a Python function invokes the GIL which kills parallelization and this is an inherent limitation of Python. That said, I try not to use Python functions these days. I write transformations in SQL (or native expressions in Polars) and these can be executed at full speed with complete vectorization and parallelization.
- dr_kiszonka 4y agoThanks, wenc - I will give loading csvs with polars a try! Moving to SQL sounds like a good idea. I just don't have the time to convert my codebase and configure everything correctly.
- hn2017 4y agowould things like "to_sql" be faster in Polars too?