4 ms·
> People typically start with Excel and graduate to Pandas somewhere in the GB range. Pandas serves them well into the 10s of GBs range, and then they start hit
by sjtrny 24d ago
> People typically start with Excel and graduate to Pandas somewhere in the GB range. Pandas serves them well into the 10s of GBs range, and then they start hitting memory issues, slow computation, or become frustrated with Pandas’ baroque API.
Assumes that a project moves beyond 10s of GBs. I guess 99.9% of projects that import pandas fall well below this threshold.
- minimaxir 24d agoAt my work I had convinced the ML pipeline engineers to switch from pandas to polars for even small ETL pipelines and there were notable performance gain with better CPU/memory utilization. If a library is performant at large datasets, it is likely performant at small ones too.
- sjtrny 24d agoI’m not disagreeing with that statement at all. You missed my point that there are thousands of people making small Python scripts for education and personal projects everyday. In those circumstances the performance concerns are irrelevant and the ergonomics of good pandas documentation and community knowledge make it a better choice.
- minimaxir 24d agoThat inertia is not a good thing, and it's partially why there's stagnation in data science. Polars is more than mature enough in both documentation and resources for it to be a daily driver.
- __eddie__ 24d agoTrue, but also if Polars and DuckDB offer a similar experience with the ability to scale beyond that range, why not use them (for new projects)? This isn't a call to arms to rewrite everything in the new shiny, just consider the new shiny for new shiny things
- appplication 24d agoNot to mention less footguns. I used to spend my days unwinding bad habits DS pick up from years of panda abuse. Credit to pandas for popularizing dataframes in Python, but polars and duckdb are objectively better APIs in addition to their implementation improvements. Agree that it’s time we let it go.
- __eddie__ 24d agoYes, this was something I emphasised more in the slides / my actual talk. But it's just significantly easier to reason about the APIs for both DuckDB and Polars
- __mharrison__ 23d agoMost folks just need to learn how to use pandas well and that will open enough doors. Then they can move to polars or duck if needed.