4 ms·

There are so many dataframe libraries, many of which have APIs closely following pandas, but not drop-in replacements. I wish we could agree on a standard descr

by abeppu 5y ago

There are so many dataframe libraries, many of which have APIs closely following pandas, but not drop-in replacements. I wish we could agree on a standard describing the core parts of what a dataframe must do, such that code depending only on those operations can easily move between dataframes.

ddavis 5y ago
There is an effort for this: https://github.com/data-apis/dataframe-api https://github.com/data-apis/dataframe-api
- eternalban 5y ago
  https://data-apis.org/dataframe-protocol/latest/design_requirements.html https://data-apis.org/dataframe-protocol/latest/design_requi...
teruakohatu 5y ago
Worse than that, pandas has a terrible API to start with. Going from the QueryVerse to Pandas feels like going back in time.
- bshipp 5y ago
  I've used pandas off and on for the better of a seven or eight years and, unlike other python libraries, I always feel like I'm starting from scratch every time I begin a new project. following a tutorial/official API on a second screen just to remember how to do fairly basic stuff. The reason is that once I'm done building whatever model I've needed it works so well I don't have to touch it again for a few years and I forget everything I learned (or the API changes again).
sdfgsdf 5y ago
In Julia there's something better, called Tables.jl. It's not exactly an API for dataframes (what would be point the of that? You don't need many implementations of dataframes, you just need one great one). Instead it's an API for table-shaped data. Dataframes are containers for table-shaped data.
devin-petersohn 5y ago
This was my PhD focus. We identified a core "dataframe algebra"[1] that encompasses all of pandas (and R/S data.frames): a total of 16 operators that cover all 600+ operators of pandas. What you describe was exactly our aim. It turns out there are a lot of operators that are really easy to support and make fast, and that gets you about 60% or so of the way to supporting all of pandas. Then there are really complex operators that may alter the schema in a way that is undeterminable before the operation is carried out (think a row-wise or column-wise `df.apply`). The flexibility that pandas offers is something we were able to express mathematically, and with that math we can start to optimize the dataframe holistically, rather than chipping away at small parts of pandas that are embarrassingly parallel. Most dataframe libraries cannot architecturally support the entire dataframe algebra and data model because they are optimized for specific use-cases (which is not a bad thing). It can be frustrating for users who may have no idea what they can do with a given tool just because it is called "dataframe", but I don't know how to fix that. [1] https://arxiv.org/pdf/2001.00888 https://arxiv.org/pdf/2001.00888
- carterschonwald 5y ago
  This is really cool! Thx for sharing
- cardosof 5y ago
  Awesome work, thanks!
gpderetta 5y ago
I believe Codd took a stab at it a few years ago. He had some success, but didn't break in data science.
- ivirshup 5y ago
  The guy who coined a new term every time he had a new product to sell?
contravariant 5y ago
That's SQL isn't it?
- gpderetta 5y ago
  Well, relational algebra/calculus, but close enough.
chrisaycock 5y ago
Types for Tables was posted to HN last week: https://news.ycombinator.com/item?id=29509439 https://news.ycombinator.com/item?id=29509439 They have a benchmark for expressiveness (as opposed to performance). Part of this inquiry has been to form a "standard library" of Dataframes operations.
austospumanto 5y ago
https://github.com/austospumanto/minimal-pandas-api-for-polars https://github.com/austospumanto/minimal-pandas-api-for-pola... pip install minimal-pandas-api-for-polars I wrote a library that wraps polars DataFrame and Series objects to allow you to use them with the same syntax as with pandas DataFrame and Series objects. The goal is not to be a replacement for polars' objects and syntax, but rather to (1) Allow you to provide (wrapped) polars objects as arguments to existing functions in your codebase that expect pandas objects and (2) Allow you to continue writing code (especially EDA in notebooks) using the pandas syntax you know and (maybe) love while you're still learning the polars syntax, but with the underlying objects being all-polars. All methods of polars' objects are still available, allowing you to interweave pandas syntax and polars syntax when working with MppFrame and MppSeries objects. Furthermore, the goal should always be to transition away from this library over time, as the LazyFrame optimizations offered by polars can never be fully taken advantage of when using pandas-based syntax (as far as I can tell). In the meantime, the code in this library has allowed me to transition my company's pandas-centric code to polars-centric code more quickly, which has led to significant speedups and memory savings even without being able to take full advantage of polars' lazy evaluation. To be clear, these gains have been observed both when working in notebooks in development and when deployed in production API backends / data pipelines. I'm personally just adding methods to the MppFrame and MppSeries objects whenever I try to use pandas syntax and get AttributeErrors.

by abeppu 5y ago

ddavis 5y ago

There is an effort for this: https://github.com/data-apis/dataframe-api https://github.com/data-apis/dataframe-api

eternalban 5y ago
https://data-apis.org/dataframe-protocol/latest/design_requirements.html https://data-apis.org/dataframe-protocol/latest/design_requi...

teruakohatu 5y ago

Worse than that, pandas has a terrible API to start with. Going from the QueryVerse to Pandas feels like going back in time.

bshipp 5y ago
I've used pandas off and on for the better of a seven or eight years and, unlike other python libraries, I always feel like I'm starting from scratch every time I begin a new project. following a tutorial/official API on a second screen just to remember how to do fairly basic stuff. The reason is that once I'm done building whatever model I've needed it works so well I don't have to touch it again for a few years and I forget everything I learned (or the API changes again).

sdfgsdf 5y ago

In Julia there's something better, called Tables.jl. It's not exactly an API for dataframes (what would be point the of that? You don't need many implementations of dataframes, you just need one great one). Instead it's an API for table-shaped data. Dataframes are containers for table-shaped data.

devin-petersohn 5y ago

This was my PhD focus. We identified a core "dataframe algebra"[1] that encompasses all of pandas (and R/S data.frames): a total of 16 operators that cover all 600+ operators of pandas. What you describe was exactly our aim. It turns out there are a lot of operators that are really easy to support and make fast, and that gets you about 60% or so of the way to supporting all of pandas. Then there are really complex operators that may alter the schema in a way that is undeterminable before the operation is carried out (think a row-wise or column-wise `df.apply`). The flexibility that pandas offers is something we were able to express mathematically, and with that math we can start to optimize the dataframe holistically, rather than chipping away at small parts of pandas that are embarrassingly parallel. Most dataframe libraries cannot architecturally support the entire dataframe algebra and data model because they are optimized for specific use-cases (which is not a bad thing). It can be frustrating for users who may have no idea what they can do with a given tool just because it is called "dataframe", but I don't know how to fix that. [1] https://arxiv.org/pdf/2001.00888 https://arxiv.org/pdf/2001.00888

carterschonwald 5y ago
This is really cool! Thx for sharing
cardosof 5y ago
Awesome work, thanks!

gpderetta 5y ago

I believe Codd took a stab at it a few years ago. He had some success, but didn't break in data science.

ivirshup 5y ago
The guy who coined a new term every time he had a new product to sell?

contravariant 5y ago

That's SQL isn't it?

gpderetta 5y ago
Well, relational algebra/calculus, but close enough.

chrisaycock 5y ago

Types for Tables was posted to HN last week: https://news.ycombinator.com/item?id=29509439 https://news.ycombinator.com/item?id=29509439 They have a benchmark for expressiveness (as opposed to performance). Part of this inquiry has been to form a "standard library" of Dataframes operations.

austospumanto 5y ago

https://github.com/austospumanto/minimal-pandas-api-for-polars https://github.com/austospumanto/minimal-pandas-api-for-pola... pip install minimal-pandas-api-for-polars I wrote a library that wraps polars DataFrame and Series objects to allow you to use them with the same syntax as with pandas DataFrame and Series objects. The goal is not to be a replacement for polars' objects and syntax, but rather to (1) Allow you to provide (wrapped) polars objects as arguments to existing functions in your codebase that expect pandas objects and (2) Allow you to continue writing code (especially EDA in notebooks) using the pandas syntax you know and (maybe) love while you're still learning the polars syntax, but with the underlying objects being all-polars. All methods of polars' objects are still available, allowing you to interweave pandas syntax and polars syntax when working with MppFrame and MppSeries objects. Furthermore, the goal should always be to transition away from this library over time, as the LazyFrame optimizations offered by polars can never be fully taken advantage of when using pandas-based syntax (as far as I can tell). In the meantime, the code in this library has allowed me to transition my company's pandas-centric code to polars-centric code more quickly, which has led to significant speedups and memory savings even without being able to take full advantage of polars' lazy evaluation. To be clear, these gains have been observed both when working in notebooks in development and when deployed in production API backends / data pipelines. I'm personally just adding methods to the MppFrame and MppSeries objects whenever I try to use pandas syntax and get AttributeErrors.