3 ms·

Still waiting for Wes to admit that the pandas API is a mess and support the development and adoption of a more dplyr-like Python library. Pandas was a great s

Still waiting for Wes to admit that the pandas API is a mess and support the development and adoption of a more dplyr-like Python library. Pandas was a great step on the path to making Python a decent data analysis language, Wes is smarter than me, I never could have built it, but it's time to move on.

closed 4y ago
Hey--I maintain a port of dplyr to python, called siuba[1]! Right now it supports a decent number of verbs + SQL generation. I tried to break down why R users find pandas difficult in an RStudioConf talk last year[2]. Between siuba and tools like polars and duckdb, I'm hopeful that someone hits the data analysis sweet spot for python in the next couple years. [1]: http://github.com/machow/siuba http://github.com/machow/siuba [2]: https://youtu.be/w4Mi0u4urbQ https://youtu.be/w4Mi0u4urbQ
- sebastiansm 4y ago
  I learned data analysis in the Hadley/Tidyverse way and constantly struggle when working with pandas. I'll try siuba this week at work. Just one question, this runs over pandas? It's possible to get the pandas syntax like dbplyr gives the SQL query syntax?
  - closed 4y ago
    Yeah, it runs over pandas or sql databases! Here's an example querying sql: https://siuba.readthedocs.io/en/latest/intro.html#Working-with-SQL https://siuba.readthedocs.io/en/latest/intro.html#Working-wi... You can use the verbs collect() and show_query() like in dbplyr. If you DM me on twitter, would love to set up time to hear about your work / walk through siuba! https://twitter.com/chowthedog https://twitter.com/chowthedog
RobinL 4y ago
I think that's part of the point of his current project: Arrow. Specifically, one of its goals is to implement the low level computations, whilst enabling more competition in the space of data analysis API design.
wesm 4y ago
If you read my slide decks over the last 7 years or so (while I've been working actively on Arrow and sibling projects like Ibis) I've been saying exactly this. See e.g. https://ibis-project.org/ https://ibis-project.org/
targafarian 4y ago
Why wait when you can read years old blog posts on his personal blog talking about lessons learned (mistakes made) in this iteration of Pandas. You're "waiting" for him to personally call you or something?
- hackernewds 4y ago
  He did personally reply on the thread above - which was really gracious of him.
imcoconut 4y ago
He has been saying exactly this for many years and has lead many efforts to improve both the implementation and API via projects like arrow and pyarrow. Also, pandas was purpose built for a pretty specific domain of financial timeseries and cross-sectional data analysis at a time when the python ecosystem was much younger and very different from today. It's not his fault it became so wildly successful! (actually it was - it's a great piece of sofware :))

closed 4y ago

Hey--I maintain a port of dplyr to python, called siuba[1]! Right now it supports a decent number of verbs + SQL generation. I tried to break down why R users find pandas difficult in an RStudioConf talk last year[2]. Between siuba and tools like polars and duckdb, I'm hopeful that someone hits the data analysis sweet spot for python in the next couple years. [1]: http://github.com/machow/siuba http://github.com/machow/siuba [2]: https://youtu.be/w4Mi0u4urbQ https://youtu.be/w4Mi0u4urbQ

sebastiansm 4y ago
I learned data analysis in the Hadley/Tidyverse way and constantly struggle when working with pandas. I'll try siuba this week at work. Just one question, this runs over pandas? It's possible to get the pandas syntax like dbplyr gives the SQL query syntax?
- closed 4y ago
  Yeah, it runs over pandas or sql databases! Here's an example querying sql: https://siuba.readthedocs.io/en/latest/intro.html#Working-with-SQL https://siuba.readthedocs.io/en/latest/intro.html#Working-wi... You can use the verbs collect() and show_query() like in dbplyr. If you DM me on twitter, would love to set up time to hear about your work / walk through siuba! https://twitter.com/chowthedog https://twitter.com/chowthedog

RobinL 4y ago

I think that's part of the point of his current project: Arrow. Specifically, one of its goals is to implement the low level computations, whilst enabling more competition in the space of data analysis API design.

wesm 4y ago

If you read my slide decks over the last 7 years or so (while I've been working actively on Arrow and sibling projects like Ibis) I've been saying exactly this. See e.g. https://ibis-project.org/ https://ibis-project.org/

targafarian 4y ago

Why wait when you can read years old blog posts on his personal blog talking about lessons learned (mistakes made) in this iteration of Pandas. You're "waiting" for him to personally call you or something?

hackernewds 4y ago
He did personally reply on the thread above - which was really gracious of him.

imcoconut 4y ago

He has been saying exactly this for many years and has lead many efforts to improve both the implementation and API via projects like arrow and pyarrow. Also, pandas was purpose built for a pretty specific domain of financial timeseries and cross-sectional data analysis at a time when the python ecosystem was much younger and very different from today. It's not his fault it became so wildly successful! (actually it was - it's a great piece of sofware :))