4 ms·
The places where pandas blows SQL out of the water are in exploratory data analysis and visual output. The exploratory data analysis part is mostly a product o
by wswope 5y ago
The places where pandas blows SQL out of the water are in exploratory data analysis and visual output.
The exploratory data analysis part is mostly a product of composability and verbosity. Composability is hard to define succinctly, but an easy example of it would just be the ability to easily define a filter as a variable and then reference that same filter in later code. Python provides a better interface for defining higher-order operations than pgPL/SQL, hands down. As for verbosity, I get data requests at work that require a lot of chained transformations - where I’d have to write 5-10 CTEs and a bunch of GROUP BYs and window functions that would total ~300 lines of SQL. Pandas equivalent would be <50 lines, and also easier to iterate on due to the composability factor. Other “killer-feature” examples of pandas terseness: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.pivot_table.html https://pandas.pydata.org/pandas-docs/stable/reference/api/p...,
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.value_counts.html https://pandas.pydata.org/pandas-docs/stable/reference/api/p....
Feature parity varies by db, but pandas is generally much more expressive and capable than SQL for string munging and time-series stuff.
This is before getting to Jupyter, which blows every SQL interface I’ve ever seen out of the water. One-liner plotting with Pandas is a complete game changer for investigating any unfamiliar dataset; the first class plotting integration is really hard to beat.
All that said, can’t comment on your coworker’s situation without knowing the context. A billion time series datapoints might be fine, but if it’s a wide table using 100s of gigs, probably not smart. I wouldn’t use pandas in a large ETL pipeline without a very good reason, nor would I ever use it for serious joining. If he is just straight up trying trying to functionally replace SQL with pandas code, he’s definitely not doing it right.
(Am data engineer, have been working professionally w/ SQL+Pandas on a near-daily basis for over 3 years.)