50 ms·
Strongly disagree with the author. For dumb simple select group by OLAP queries on medium data ? Sure clickhouse local or duckdb works perfectly well. But if
by bijowo1676 14d ago
Strongly disagree with the author.
For dumb simple select group by OLAP queries on medium data ? Sure clickhouse local or duckdb works perfectly well.
But if you need to construct dataset ? Or process existing dataset, do heavy filtering, transformation, reshaping, splitting? The proper ETL work, then pandas is really the perfect use case.
And pandas can work with small memory footprint as well, its actually trivial to do that, plus there are libraries like Modin that are upgrades over pandas with pandas api
Pandas is the swiss knife tool of data science that lets you do anything with the data and it integrates well with ML libraries
- minimaxir 14d agoIt's subjective, but IMO polars's APIs are better and than pandas's for ETL. Notably, pandas ported over polars's use of .col() for the 3.0.0 release: https://pandas.pydata.org/docs/whatsnew/v3.0.0.html#initial-support-for-pd-col-syntax-to-create-expressions https://pandas.pydata.org/docs/whatsnew/v3.0.0.html#initial-... Also, pandas's group_by() and window function pipelines made me want to tear my hair out while both are easy in polars.
- __mharrison__ 14d agoUse pandas if you want advanced analytics, visualization, or ml. Use SQL if you need to move data around.
- bijowo1676 14d agoHow would you use sql to move data from three excel files (some of them may have arbitrary number of worksheets), pack of json files, two csv files, one mysql, one postgres db, some parquets, some of them in S3) ??? And in the end save processed data in aws s3 in another format like iceberg or whatever In pandas a lot of these are one-liners that are impossible in sql (depends on what sql engine you have?)
- __mharrison__ 14d agoThose are generally rules of thumbs for data pipelines. (Plus duckdb can read Excel these days I think).